Why did you agree to something you don't want? You deserve what you want in life, not whatever crumbs someone wants to throw you, so don't settle.
I think your first steps should be to do whatever you can to build your self-esteem, state clearly and directly what you want and expect from the relationship, and be prepared to walk away if he wants something different or tries to make you feel fucked-up or wrong in some way for "holding him back."
The world opens up... when you do.
Oh, oh, can't you see? Love is the drug for me. ~Bryan Ferry
"Love is that condition in which another person's happiness is essential to your own." ~Robert Heinlein