OUCH! Those are some not so nice things to say to a person. I feel for you.
Maca and I have had some struggles with that ourselves. I didn't go to a poly-lifestyle in a very nice way.
I had an affair.
Of course everyone said the way to fix our marriage was to get rid of GG and even if we were going to be poly now-it shouldn't include him.
But the reality is that I love both of them, and our kids love both of them-so that wasn't a functional or realistic option.
Fortunately we were all three willing to start busting our butts to find a solution that was more functional for ALL of us (us 3 and 4 kids). That has had it's up and downs, but overall it's been moving up.