The scenario you describe is why swinging makes no sense to me.
If you want to be poly rather than only swinging, why can't you continue your relationship with B? Why would his wife's no longer wanting to be with your husband affect your connection with B? It's puzzling. I just don't get how you think that everyone all has to be involved equally with each other. Poly, as you can see from reading these boards, can be manifest in numerous ways. Can you explain more?
Oh, yes, and welcome!