I have to say, the whole marriage in trouble add more people is always a red flag around these parts. For my own part, I say you need to make a decision on your marriage. There are things people will just not put up with. For some people one cheating, one instance of physical abuse, one instance of sexual abuse and it's done. Over. For others, there's a chance to make it work. IF they both want to.
Something like that should have been a wake up call for your husband. I understand loving him and wanting to make it work, however if you want him to go to therapy, alone or together or both, and he won't. Then he has given you his answer. He doesn't want to work on the marriage. He doesn't want to 'make it work'. He wants it to go away and not deal with it. So forget the poly question, what you should be doing is wondering if the marriage question is valid.
I would tell him straight out, "Honey I love you, I want to make this marriage work and am willing to put in the work to do that. I need you to do the same. That means, I think we should see a therapist."
Me: Late 30s pansexual poly.
DH: My husband of 19 yrs and father of 3 teen girls.
DC: LDR of +4 year