What does it mean when you say, "he wants to join her family to mine?" What does he envision? You said he'd love for you to live together. Well, what if you just say, "No"? You do have a choice. He does not hold all the power in your relationship. Your heart may be breaking, but that does not mean you should accept what you cannot handle just to please him. Do not let him make you believe that you have to walk. You can always say, "This is what I need. This is what I'm willing to do. This is what I'm not willing to do." He can walk, if he chooses to. But you owe it to yourself and him to stand your emotional ground and work things out honestly.
I would gently suggest that sharing someone emotionally does not go against marriage. Marriage is commitment, but that commitment does not have to take the form you were taught from childhood.
Plenty of poly relationships exist in which partners do not live together, and you have the right to expect your spouse to meet his responsibilities to you, the children, the household.
Can it work? It can. Will it work? There is really know way to know. All relationships are filled with risks.