All I can say is that your husband needs to get it in his head that he doesn't own you just because he married you.
I never understand it whenever I read (and it happens often) that a couple gets into swinging because the husband is okay with his wife sharing her body with someone else, but will not tolerate her sharing her heart. To me, that indicates an enormous devaluing of women as whole people! It says, to me, that the man sees himself as owner of a woman and that he feels entitled to dictate that she let her body be used by whomever wants it, but yet she doesn't deserve all the love she can find in this world -- real, genuine love that comes her way. Claiming ownership or possession over someone is an obsession based on insecurity, and isn't very loving at all.