This man is being honest with you. He does not want or is not ready for a relationship.
You are exactly right that this is an unhealthy relationship. From what you write, the man is using you, nothing more.
For your own protection, drop him like a hot rock. Now.
Here's how to protect yourself. Text him a message along the lines of:
"Dear A-hole: I deserve better treatment than you give me, and I have decided to invest the time I used to spend with you into finding a partner who is worthy of me. If sometime in the future you decide you're grown up enough to want a real, grown-up relationship, drop me a postcard. There's an outside chance I might still be available. Until then, though, lose my number because I'm blocking your calls. Thanks, and c u L8r. Much, much L8r."
Once you've sent the text, block his calls. This may be all you need to do, because he clearly cares very little for you and has little or no respect for you or for himself. HOWEVER, it is possible that showing him the gate may wake him up, and he may decide suddenly that you are the most desirable creature on the planet. If this happens, DO NOT FALL FOR IT! If he calls you on another number, don't answer. If he shows up at your house, don't open the door. If he won't go away, call the cops.
I'm not kidding. From what you write, this guy has serious issues, and you're not strong enough to weather the storm. Get out now.