Apart from saying you and your partner can't give each other up you haven't really said how he feels about the situation (i.e. his wife being so obviously unhappy). I think the happiness of his wife makes your position very precarious, especially if children are involved.
Polyamory is hard enough without animosity between metamours. Having said that I do know of a Vee relationship where there is no connection or friendship. The two women have their own residences and the guy, who is the hinge divides his time between the two of them as well as having his own house (although god knows when he has time to live in it). They do all live in different cities so there is no problem with 'coming out' as you put it. Each couple acts independently when together. They manage but struggle.