Oh how I can relate! Here's the thing, everyone has an opinion. His friends care about him and are worried he is entering into something that isn't healthy for him.
Karma, Cricket and I went through this on a lot of different levels. People thought Karma was using both of us, people thought I just wasn't strong enough to leave my husband who obviously couldn't be happy with just one woman, Cricket was a home wrecker, it was a huge mess.
I got so sick of people who weren't living my life having an opinion. But I realized it was because they cared and didn't understand. Once explained that I was happy, Karma was happy, Cricket was happy, no one was doing anything to harm the other, people lightened up.
One thing that helped was explaining out rules. Which are really simple- Happy, Healthy, Sane. If any of these aren't happening then it's time to re-evaluate the situation.
For me, as long as WE are happy, healthy and sane, what more could we ask for? And once that is explained to those who love us, they seem to accept things and move on.
I know it can be frustrating because it seems like their concerns are a reflection on you, but they don't know you, so don't worry about it. Be yourself and if they come around great, if not, it sucks, but there isn't much else you can do.