Well you're right to be concerned because when you're talking triads, the "weakest link" can often set the tone for entire triadic relationship. Been there, done that. Fortunately though it's pretty clear you love both your partners and both love you. So fortunately it's only a sexual link that's got problems, not a love or romance one. Therefore talking about it as something that could ruin the whole thing and make you leave is definitely over the top---so I would relax on that point.
You've talked to him and he's patient. That's a good sign!
At this point what you really need to do is get to the root of this problem. I would highly recommend at this point finding a gay-friendly and poly-friendly sex therapist. It may take a professional knowledge to figure out exactly why this is happening. (And this advice is coming from someone who normally shies away from jumping to therapy as the cure for much....I've never cared for it that much myself.)
Keep the waves of communication open and talk to both your partners about your feelings, wants, etc and be sure to keep your emotional connection strong, even if there are bumps in the road on the physical connection. Hopefully with some work and maybe some professional help, it can be figured out and fixed!
We're rooting for ya!