your husband is acting out of the mindset that comes with swinging... DON'T FALL IN LOVE DAMN IT! BAD BAD BAD! swinging is very monogamy based.. There is a lot of control of emotions and while there are no sexual boundaries. There is a lot of emotional, ownership, and control boundaries... poly is much different.
Swinging is awesome if you want to have a loving couple relationship and have lots of fun times having sex with others. It really works for some people and is a huge benefit to the "couple" relationship... some people just are not able to control themselves emotionally after a time and, tadum... fall in love, start caring for someone and start wanting more than just sex. This seems to be where you are, yet your husband is not.
The transformation between mono and poly is a huge one. It's a 180 degree change of view on how relationships are conducted and some people just can't make that change or refuse to. That is just who they are and their right. The thing is that the person needing the change is often left hanging and in limbo for ages and ages until a change happens or the relationship ends. How long that lasts is up to you... if you leave it, you could end up depressed, angry, bitter and resentful... without love for anyone, anything or the desire to love. Someone controlling you will do that in my belief.
You could leave and find like minded people to befriend and date, and leave your relationship behind. You could also keep at it and not let down for a moment that you are ready to do what it takes to ensure he doesn't feel threatened by other loves, that he has enough time with you and to prove that this is really going to be a more heightened connection, bond, loving and fulfilled relationship if you and he do what it takes to stay together... Really, you can't deny that this will be make or break time sooner or later... if not because of this man, then someone else.
Has he read here? Maybe that would be a start. Maybe seeking out a local poly group to go to... maybe the therapist will help... whatever works to move the two of you forward... even if you stick to the agreement that you don't talk to this other man, but work on your relationship instead... all is negotiable and will work if he is open to it and ready... the big thing is that love is timeless. There is lots of time and no rush. When the time is right and everyone is ready, then someone new in a relationship dynamic becomes a blessing and a gift. Preparing is the only way to get there...
Anyone want to be friends on Facebook?
Send me your name via PM