I agree totally with frisky.
Most right thinking couples (or triads or quads, or whatever) set certain hardlines when it comes to outside relationships. I definitely wouldn't suggest the childish "This is your problem not mine, NYAH!" approach. It's still possible to keep both relationship from this, but it will take a lighter hand. Outright refusal of your husband's demands will make him resist more or straight-out divorce you.
Sometimes the exertion of control helps a partner feel like they HAVE control. If your husband feels comfortable that he is your primary, you might notice this problem fix itself. Counseling might help a lot with that, it might not. Being resistant and forcing your husband on his ultimatum, that's definitely not going to help unless you want him to walk away.