I'm of the opinion that anyone that is actively, consciously disrespectful of me, my partner, our relationship or our home has no place in our lives. That goes for friends AND family.
What this really boils down to is that if someone is a jerk, we call them on it and give them another chance. Continuing that pattern of behavior is unacceptable and cause for ending of relationships. I'm too old/busy/sane for that shit, and I have zero patience for it after it's been addressed.
If that's controlling, so be it. But I too have abandoned my local poly community due to judgement and harassment. It sucks, but that's why we're here, right?