Confused , I am gonna take a different tack to what others are saying and say something that I have a gut feeling about on this one.
I think you are with the wrong person.
I don't think this man will EVER make you feel good about yourself. This does happen, we make mistakes. There are lots of lovely people out there for you, mono or poly.
You deserve a chance at better.
Some others might feel this reply is less than useful for you, but I'm just being honest.
Good luck with your life.
"The secret of life, though, is to fall seven times but to get up eight times"