Oh, I'm so sorry to hear that! At least you have J to help you through this, but I know it's rough. Your hubby has flip-flopped back and forth so many times, maybe he will come around again. But such a roller-coaster for you!
Odd that you two were swingers before, and he's got another squeeze, but now he calls you a whore for loving another beside him. I've never understood the swinger mindset, but something is not getting through to him. I don't recall if he's in therapy, but it might help to start that, if he's not. Maybe it would help you, too.
I hope you and DH keep talking and trying to work it out. Has he ever come here to read this forum? Might also be helpful for him.
Independent solo polyamorist seeking lover-friends willing to invest in friendship, companionship, and love, but without a need for partnership.
Never confuse commitment with exclusivity, love with ownership, nor sex with intimacy!
For me, it is far better to grasp the Universe as it really is than to persist in delusion, however satisfying and reassuring.
~ Carl Sagan