AC, this is a tough situation!
Why did you marry him in the first place? I get the sense it was because he was committed to you, and you saw him as a steady stable guy who was a good provider and had the potential to be a good dad. All very well.
But you are missing being with a guy who gets turned on just by being near you, who obviously desires you and actually lets you know it. I feel bad he never puts the moves on you and you always have to initiate sexual relations.
So, he has low to no desire for sex in general and you don't feel all pretty and desirable around him. Men who actually desire you (as well as caring for you deeply emotionally) are a huge temptation.
I get how you feel like a perv for ever having desire for others. My ex used to try and control me in this way as well. You say your h said he'd leave you eventually if you had other relationships, but what about you? Do you fantasize about leaving his emotionless non-sexy ass as well sometimes?
Love withers under constraint; its very essence is liberty. It is compatible neither with envy, jealousy or fear. It is there most pure, perfect and unlimited when its votaries live in confidence, equality and unreserve. -- Shelley
Mags (poly, F, 60) loving miss pixi (poly, F, 38) since January 2009, living together since 2013
also loving Punk (monogamish, former swinger, 42, M) since Oct 2015
"Master," (mono, 34), miss pixi's Dom for 2 years