I'm not sure what to add so it doesn't sound like I'm creating my own pity party, lol.
I've been together with my wife for 10 years, married 7 years and newly poly for maybe 2 months. Aside from 1 week of bliss at the thought we might hit it off with an amazing couple it's been mostly hell for me as a male it turned out the male and my wife hit it off. The woman was not interested in me so the issues my wife and I previously had at home have just intensified like 10 fold and I'm at a loss how far I'm willing to drop my standards since it seems impossible to find any women into a "loving relationship" who will accept a married male.
Where with casual sex, unhappy married women, etc it seems most have no problem with married men and some actually prefer it thinking would be safer than a guy who sleeps around with everyone.
There are more experienced members here who are almost like counselors, lol. All I can think of is figuring out both your feelings, if anything you've done lately made him more insecure to want to put a stop to your poly relationship.
I know a issue my wife and I have is her is the physical bond at home since I'm more into cuddling, touching and being intimate in that way. Hers is more verbal and "acts of service" according to this test below. So we at times both feel the other is ignore our needs being met.
I'm not sure the type of guy he is, if you can simply sit down and talk as adults as to what is bothering him. How you can meet his needs and what can you do to have him continue to be comfortable with you continuing your poly relationship.