Ok, you need to view these relationships separately. Firstly, your marriage. Are you still compatible with your husband? Do you want to be with him? Could you sacrifice your desire for polyamory for a happier marriage? Do you want to swing?
Then B. He seems to have his own issues right now. Experienced poly people are usually a little bit better at not letting emotions from one relationship flood the other in a way that negatively impacts that other relationship. Regardless of whether you do get back together one day, I think you should focus on getting yourself in a better place. That will help any relationship you are in.
Please forward any comments, complaints or suggestions about my various thoughts and opinions here