Hi and welcome!
Not really familiar with the swinging scene, but is the problem that you don't want to have a relationship with B if N is not going to continue her relationship with your husband?
Why is it imperative that you and husband always share? You can't force connections, and can't have pity sex just to prevent someone feeling left out, like N probably has discovered.
Why can't your husband have another relationship with another female independent of you if he wants? Since people couple up for so many different reasons, it seems rather unlikely that two couples will form equally strong connections and attractions amongst them that will last equally long.
Should you have seen this coming? Um, the premises you started with seem rather unrealistic, so maybe it's more about how you should have considered that not everyone will want to play and love together all the time for ever.
Me: bi female in my twenties