I just keep thinking that I'm in the wrong, and if I could just get to where he is with the poly stuff, we wouldn't have any problems. He gives me full support and autonomy to pursue anything I want. This is literally the only thing we fight about, but obviously, the fights are bad.
You are not in the wrong. Though he may have messed up your head. Consider taking highlighter to the tactics at speak out loud.
You keep flipping back and forth between seeing how terrible this is, and trying to minimize his badness. You also minimize your own hurt and your own importance.
Do not play that down like it's a minor thing. This is not a quibble over Crest vs Colgate and just going "To each their own toothpaste!" This is core values stuff -- he does NOT respect your boundaries or honor your saying NO.
This is NOT ethical polyamory.
Also, if anyone can weigh in on this... I don't know how to deal with the friend who unknowingly started this all.
Leave her unknowing. It is not her business. Focus your energies on getting YOU to a healing place. Not spreading it thin on side issues. You can always reconnect with her LATER. Unless you plan to ask her for help in getting you out of this or something... it's just another side distraction.
Be honest with yourself here. You are unhappy in this relationship.
Why do you want to pretend to someone everything is ok when it isn't?
Stop avoiding. Do what you need to do to get OUT.