There is nothing wrong with the way you feel. I feel as you do and have agreed with Mono not to pursue other men sexually. I have a non-sexual boyfriend as a result and you know? I am just fine with that. I have found that it is the close emotional bond I was craving, not the sex. Part of that for me is that I get all my sexual needs met quite nicely and don't want/need for any more. It sounds like this is where you and I differ as you have said you are not.
I wonder if it is possible that you two could work on your sex life together. If he thinks that sex outside of marriage is wrong and would not stay with you if you ventured into sexual relationships with others then how would he feel about working on your sex life and you having close relationships with the other men in your life? Maybe there is a chance that you and he could find a compromise.
I have written a lot here about compromise as a never ending teetor-totter of attempting to find a boundary agreement that works for both, but really can never be reached. Sometimes it is just like that and one has to let go of ever feeling entirely comfortable with the situation. It is possible to do that but it takes easing up slowly to the ever illusive boundary and seeing what results.
For me the illusive boundary was crossed when I got to close to Leo and Mono had a really hard time with it. He saw it as cheating, and to me it was within the bounds, just barely. We now know that the boundary was in a different place than I realized, but we are okay and it was a learning experience. Not worth ending everything... we decided to take it as a learning experience and move on.... it is possible to do that.
Anyway, I wrote a lot about that last year in my blog around Dec 2010 if you want to read about it... please PM if you want too
I'm not saying that you shouldn't be done with this situation and move on, because I still see some options... if you and he love each other, work well together and are willing to do some work... at the end of the day I would not begrudge your choice to stay or go, just so you know. I know all too well how much work it is that is before you and ending your marriage is also a viable option.