Only you can decide at what point to stop living your life a certain way because of what other people think. Yes i realize that is easier said than done, and there are all sorts of ramifications and repercussions that come with telling the world to eat a big steaming pile of "idgaf". We all face choices that expand and limit our horizons in different ways. For example, you have kids and a job, it's not like you can just say "ok, let's all move to Boston or Seattle where people won't be all up in our private personsl lives". But you can think about maybe that's something to work toward over the next 5 years, if you're one of those "5 year plan" types.
Sorry i don't have the magic bullet for you. Maybe someone else does.