It seems to me that you do not have a problem sharing a man with another man. Your mood, self-esteem, and position are not threatened by that. It's when a woman comes into the picture that there begins to be all kinds of problems, rules, restrictions, etc.
What is it about women that makes them so much more threatening than men? It seems that you have some buried issues there that need to be dug up and examined.
What does your poly counsellor have to say so far?
Love means never having to say, "Put down that meat cleaver!"