During your wedding vows (i.e. commitment promises), did you use the standard "Forsaking all others"? If yes, then you explicitly made a promise to him that there would be no other in your life.
Even if you didn't, the expectation in our society is that a marriage is monogamous unless explicitly negotiated otherwise.
I think that you knew this.
So you married him under false pretenses. Your marriage started out with a lie.
Or am I (and others posting here) misunderstanding something about this?
"Listen, or your tongue will make you deaf." - Native American Proverb