First of all, swinging REQUIRES a very strong, open, trusting relationship. It doesn't sound like either of you should have been swinging.
Second, look up NRE (new relationship energy)...could this have been what you and/or B was feeling?
Now, I've been thru something very similar: After being married for years (and realizing that it was a mistake after the second but she was pregnant...), my HS girlfriend...my soulmate came back into my life. I couldn't leave until my child was old enough so we both waited to get back together. When we finally were moving in that direction and I was ready to leave for her, she became 'scared' (her word) and rushed into another relationship (she didn't want to be alone and felt like we would never be together even though, by now, we could if she wanted it). I went through the divorce and lost most everything (no regrets) thinking she would come back. We talked all of the time, but due to distance couldn't actually see each other...about everything including her messed up relationship and the day we would be together, but that never came. I finally walked away and cut off all contact with her when she was willing to be a doormat for the on-again-off-again relationship she tried to maintain (although she hasn't even tried to contact me after I quit talking with her). I was willing to give up everything for her, and in many ways I did. It was she who told me that we were always soulmates and that she never stopped thinking about me, even during the time we were apart (15 years ish). I still think about her way more than I want to and wish I could stop it.
Still, I found another woman who loves me how I am and we have created a very strong, open, trusting relationship. Things have never been better and are better than I ever thought they could be. We talk about everything with total trust. Long story short: I love her more than I ever thought it was possible to love...and because of that I also care very deeply for our (originally her) girlfriend...and her girlfriends husband.
Point is I was afraid to leave my marriage because I didn't want to be alone. I 'gave up' everything for someone who didn't feel she was worthy of having anyone sacrifice anything for her. I was afraid I had lost everything but I moved on. Things got better, WAY better. Just because things don't work out how you want them to doesn't make life less than what you wanted. Sometimes, things can be better than you ever imagined them to be.
I still want to call her and ask her why my soulmate left me.