I see a couple of options here. One of which you already said and that's so long as you're comfortable with the type of relationship that you have with an individual, does it matter what they call it? Secondly, explain to your lovers exactly what you think. That your men are your men and they are friends with benefits. (or whatever other terminology you choose)
You have too many reasons to be happy, to be so stressed out.