In that case it needs to be between her and your husband, who may be exacerbating the problem by the way he explains things.
Once the two of you have agreed to something he needs to take equal ownership of it. Does he say "I won't go to sex parties with you", or "My wife says I can't go to sex parties with you"? The former makes it something that is what he will or won't do by his own volition. The latter sets you up as a controlling bad guy by sidestepping the fact that he agreed to the situation himself. If he's taking the second option he needs to re-think his wording.