Looks like there is a fear issue behind it.
Maybe you don't feel threatened by women because you know you can bring something they can't to your partners (being a man and all) but with other men this "security" vanished?
I'm just speculating here. It could be something completely different. If that's what it is though maybe you need more reassurance from your partners. I think it's a good idea to be very open and discuss it with them, not to limit them but because they might be able to help you.
Has there been cases of specific individuals that one of your partners liked? You mentioned one that was more of a "pity date" and I'm not sure how representative that is, since the situation would feel awkward to me: you knew she wasn't actually interested in him, therefore you might have been worried for her or something like that.
Although your anxiety thing leads me to believe you might be worried for you, not them... But maybe you're actually worried for them but feeling powerless in case things go wrong? It might help to identify what exactly scares you.