Would it help if you had the right to veto some men, but not all? Maybe you could negotiate a list of reasons why you could reject someone... Or a list of characteristics men have to fit for you to accept them...
If you can agree on one, that is, if it's not totally unfair and unbalanced, maybe it will help? The point would be for you to feel secure that the men your partners are dating aren't a threat to you or them (depending on what your fears are about).
Sure, you feel like you wouldn't be okay with any men at all, but maybe you can cut it down to some characteristics? Find what would be okay?
If you really can't, then you'll need to find a solution in yourself... I'm not sure how... But maybe talking about it with a poly-friendly counsellor or therapist might help? If it's a deep-rooted issue, maybe they'll be able to pinpoint it for you. And sometimes just knowing what the problem is can be enough.
Me: 30F, straight
Seamus: My husband, 32M, straight
Dragon: My boyfriend, 29M, pansexual, married to Fox
Fox: My boyfriend, 29M, homoflexible, married to Dragon