A marriage needs two people to put in the hard work required to maintain itself. If he flat-out refuses to do any of that work, then it's a done deal.
It's not that you're choosing poly over him. It's that you're choosing yourself over your marriage.
While some kinds of needs can be met by someone else, others can't. An example of something that can be met elsewhere is sex or kink.
Intimacy is not like sex or kink. It is a need of the relationship. It is also a need of most people as individuals, but that's separate.
Poly is irrelevant to why your marriage is failing. You're with a partner who refuses to carry his share of the relationship. He is refusing to do the hard work required to maintain your relationship.
At this point, I would probably give him one last chance. Don't mention poly for a while because right now, it's not the point. The point is that your marriage requires emotional intimacy and an equal partnership. Give him one last chance, an ultimatum: You come with me for counselling to fix our broken marriage, or our broken marriage is over.
If he agrees to counselling, recover from fainting and immediately make the appointment. Get him to agree to the time. Tell him that failure to show up will be interpreted as his decision to get a divorce. Heck, even make an appointment with a divorce lawyer for the following day. Tell him to choose which appointment to keep.
Only if he agrees to counselling will it make sense to discuss poly again. Then it will definitely become part of the conversation, but recognize that you might need to put that on hold while you both work to repair the broken marriage. You're expecting him to make sacrifices to save the marriage; he'll need to know that you're willing to make your own sacrifices.
“As I am sure any cat owner will be able to tell you,
someone else putting you in a box is entirely different
from getting into a box yourself.” —bisexualbaker