It's like anything else. People will find a problem with something even if you're not "poly", so as long as whatever it is isn't illegal, let them think whatever they think and do what is best for YOU. You can't change the world, but you can change how you react to it.
That is like saying, "I will not cut my hair a certain way because some people might not accept me, or they will think I am [this way] because other people who have a green mohawk or a blond mullet "are" [this way].
It sucks when it's your parents or your boss because those people, their opinions do matter to a degree, but they can either deal with it or you can organize your life in such a way that it doesn't affect your routine.
Having said all that, I am not "practicing" a polyamorous lifestyle at this time, but I do cat rescue, and people judge me for that, so I do know what I am talking about after all.