A couple of things from your OP I think are important:
1) Your husband grew up in a different culture. I would think those cultural differences matter and need to be taken into account.
2) Your "other man" has a girlfriend and a child living at home (not clear if it is his or the gf's or theirs). What do they know about the situation? It seems there several people who would be affected by what decisions are made.
Right... My husband grew up in India, where propriety is important, and morals (on the surface, anyway) are stricter than here. Once when we were discussing how he is always putting everyone else first (the kids, me) and not standing up for his own needs, he said it is my happiness that he needs to make him happy. I jokingly asked what he would do if I found happiness sleeping with a new man each night of the week. He said, "If that's what made you happy -but I wouldn't be happy about it." Well, I don't want to make him unhappy. I think telling him I'm in love with another man would do that, and I think asking for the freedom to have sex with another man would devastate him.
As for the other man's situation -the kid is from his earlier marriage. We get along and he doesn't seem to think there is anything wrong with our friendship. The girlfriend (who doesn't live with him) is someone he has broken up with at least 3 times in the past few years and appears to be about to do so again, for reasons having nothing to do with me. I'm sure she would not want us to be as close as we are, and I like her so I feel guilty there too, but I've kind of left that for him to worry about I guess. He seems more concerned about protecting my marriage than his relationship -I don't think he's very vested in it any more. That makes him sound kind of shallow, but they have their valid reasons for trying, and for failing, to be a couple for so long. Nothing to do with me.
I also want to say that I'm really happy to have found this forum and to feel this kind of support. I admit to having confused polyamory in the past with swinging. I didn't realize how much I was ignoring my own propensity to love, and trying to shape it into something it isn't. I didn't know other people felt like this and accepted it. It's a huge relief, even though I know, with my dear husband, I am going to live a monogamous life -sexually, at least.