I understand where you are coming from. It took me awhile to settle it for myself. Basically, I found the idea of my wife/gf having sex with another woman hot and different enough that it did not feel like competition to me. But if she was with another guy, I would worry about sorts of crap that society and culture teaches/preaches. Like the guy would try to steal her. Or he would try to lay claim on her. Or he would mock me for not being able to hold on to her. I know some guys who think along those lines.
The way I dealt with it was to realize that my wife/gf would not pick someone who would cause stress in our relationship. She would not choose a jerk who would try that macho crap on her.
I also really had to think about compersion as opposed to the cuckold view. I realized I could be happy if she was happy with another guy as long as he was making her happy.
What I think helps deal with some of this is to meet the other guys in a friendly social setting. It doesn't have to be a great bonding moment, but just enough to recognize each other as a metamore. Maybe it can turn into a friendship. However, they will be real guys to you that may not seem as threatening because you will see the common everyday flaws we all have instead of an idealized guy who sounds more threatening.