I sympathise with you and I know what this is like.
Ultimately, Meera hit the nail on the head. The best revenge is living a happy and fulfilling life. They aren't there when you are snuggled up on the sofa, when you are having Earth-shattering sex, when you are warm and glowy because you've found the ideal fit in each other.
To hell with what other people say. And whilst we're on the subject of hell - I find that a good response to idiots that say these things is "only God can judge us - and judging me is blasphemy". It usually shuts people up fast.
I have spent my entire life developing a thick skin about people's opinions, but even now, at 28, of course it can be difficult. I came out as a lesbian when I was only 13 and found that the best way to put a dampener on any unwanted opinions was to answer questions as if they were almost boring to me. Matter of fact, open, blase. When people pushed it too far, I'd tell them. As adults, we need to have even more of an ability to do this. As horrible as this is for J, he needs to defend himself and your situation and speak honestly, with strength and dignity. In terms of yourself, you could always speak to L and give her a piece of your own mind - but that is entirely down to you.
If this helps at all, I completely understand what it feels like to be judged, abhorred and completely ignored by those who are meant to be close. My girlfriend's father's wife won't allow me in their house. My girlfriend's mom literally doesn't look at me, doesn't speak to me, doesn't acknowledge my voice when I try to speak to her. She stares right through me as if I literally do not exist. It's an awful feeling. Eventually, my girlfriend decided that enough was enough. She told her father that nobody, including our daughter, would be coming to their house until they got their act together - and she told her mother that they would have no contact until she found some way of dealing with it. Since then, things are still frosty, but my presence is at least acknowledged.
Sometimes, you have to stand up with the strength of your convictions. You also have to keep your head high and ignore the rest of the world.
Just remember that they care for their friend and think they are doing what's best for him. In time, they may come around. It's up to J to encourage that .
me: female, 28
GF: my girlfriend, female, 38
Hubby: my metamour, her husband, 34
2 year, open poly V relationship, LDR