Tell him you're not interested in that kind of relationship with him, you want a friendship. If he insists, tell him he's not being respectful of you. If he insists again, stop spending time with him.
It doesn't really matter why you don't want a relationship with him. If you say you don't, he should respect that.