There are very few people I let into my life or develop true connection with. There are even less that I care about thier opinions of me. I really don't care how most view my life and am more concerned about how those I love are treated. I can be isolated without great impact, it's disrespect towards my chosen family that will get a response.
In general though, I find as soon as people see us together they relax and see that we are happy. If they can't see past thier own shit and handle it, fuck them. As long as they aren't hurting my family I'm cool with not interacting with them. If thier intention is to hurt my family then I'm perfectly happy to interact with them on that level as well