If you are worried about saying the wrong thing because you are too hot-headed, is there a way you can plan to talk by IMs or something like that? You would still be talking, but typing takes longer and you would have time to see what you wrote before pressing "enter" and potentially change your mind.
I do think that talking directly is important. First, because your image of one another is skewed right now, both because you don't know each other well and because what you do know is by proxy. That needs to be fixed, as feelings you have for one another are now feelings for people who don't actually exist, but that you created for yourselves.
Secondly, because, as it was stated, it can be hard being in the middle. Not having to be the middleman would be nice for him. Also, if he's not there you'll be able to talk more freely. You might even ask her directly what he said about this or that, or if he did mention this and that. Talking about him can also make you grow closer as metamours, as sharing something like that can be wonderful.
Think about it: talking about the man you love, how much you love him, etc, with someone who doesn't think you're being silly, someone who understands, and someone who feels the same! Or if something goes wrong, having someone to go to who knows him and can tell you what she thinks he meant, and how he acted with her, and what he told her about it.
Of course, you need to trust each other first. Otherwise, you might feel like the other would be happy that things aren't going well, and wouldn't try to help.
Also, you said you felt he was on her side when asking you to talk with her... I think what he's afraid of is alienating you. You're the one distancing yourself from the two of them, he's trying to bring you back. I think it's because he cares about you, about your relationship with him. Your feelings towards her are making your relationship with him tense, and he doesn't want that, so he's trying to solve the problem. To get you to talk about it.
I think he's offering to be present for it to be more comfortable for both of you, but for the reasons stated, I think it's best if he isn't there. Plus, give him a taste of what you both got: he won't know what the two of you might be saying about him