I was just saying that i am not all that experienced to give any advise, but i feel compelled to give an outsiders point of view (however simplified).
My first reaction to your story is: it seems that you are afraid to lose the beautiful thing you have finally found, and that you are more afraid of the new couple spoiling it (your husband already proved loyalty, despite the swinging and more) whereas they haven't yet, so you want to claim (their) exclusivity?
Added to that: my personal response to 'exclusivity', no matter how much I love the person and I have full intention to perform exclusivity on all fields on my part - a part of me does get claustrofobic if the word gets too much focus/attention. Maybe the couple feels this too?
Imho (i don't have near enough relational experience, so please just see it as another point of view), i think it might be worthwile to bite less hard into this situation and give it some more air. Hard to do when you do not want to lose any of it, but it might still just be what you all need?
And I just found out it helps to focus on your own personal things (work, hobbies), instead of focusing a lot on relationships (which I do find hard: what is the perfect balance? I guess it is not static anyway, goes in waves.)