Hullo there and welcome to the forum!
I am sorry that you are going through some tough times. And in my opinion, you shouldn't just pull the plug with this other couple. It wouldn't be fair to you and it wouldn't be fair to them.
However, your husband is having conflicted emotions. Intellectually, when the situation is not actually on, he thinks he's okay and no boundaries are needed. Then when it happens, in front of him or close by, all of his negative emotions come over like a wave.
It's not right that he is punishing you or blackmailing you with threatening to end things. If I were in your situation, I would start with communicating to my hubs that it's okay to progress slow. No one is giving him extra brownie points for progressing fast. He also needs to start communicating more directly about his needs. It is not your responsibility to end interactions if he is the one having problems, no matter how many meaningful looks he gives you. He needs to start owning up to his emotions and behaviours, including the violent and abusive ones. Has he ever been to therapy with his anger management problems?
Like you said, you still have fun vanilla times. Maybe you could focus on that, and focus on building a stronger foundation with your hubs, with the focus on learning to communicate and manage emotions. If he is having 'performance problems' and is worried S might think it's her, then HE needs to talk about that to S. Spend some non-stressful, non-physical date time as a quad and in both individual couples. Of course it's important to gauge S and T's feelings as well. Are they interested in putting in the work to turn this from swinging into poly?
Me: bi female in my twenties