Yea, I kind of agree with the other's sentiments so far. You really did kind of get the cart ahead of the horse here. If you have ANY hope of salvaging this relationship you need to give it some space for awhile.
This is one of the biggest problems that can occur when starting from a swinging direction. Remember, for the vast majority of 'swingers' the cardinal rule is 'no emotional attachment'. You all (or at least you and he) have violated that rule. Not uncommon <chuckle>.
However, you are now in a VERY different world and the rules are completely different ! And it may be that none of you were aware, prepared or ready to make this crossover.
So now everyone needs to step back, take a deep breath and hit the books ! There's much you all need to know before anyone moves an inch forward !
It likely IS possible to turn this into a win for everyone but you are going to have to approach it differently. From your writing I suspect it's likely that at least his wife was firmly under the umbrella of 'no emotions'. As naive as that is - it's real. Now you ALL are going to have to delve into WHY that rule was important. And you are going to have to get your communication skills sharp as a razor to move forward. And it's going to involve some pain. For everyone. But the more kind and gentle you can be the better the process will unfold.
Remember - you are all treading in somewhat unfamiliar waters. Acknowledge that as a group. Try to make a commitment to learn TOGETHER. Try to put the emotions on low heat for the time being. If the connection between you is real - no harm can come. But you can't even be certain yet. The test of fire is just beginning.
Regardless of how this all shakes out - the journey is worth the effort. Everyone will learn and grow in ways they may never have expected. Embrace this - don't be afraid of it.
And remember, we're all here to help you any way we can.