Oh, I'm so sorry to hear that! At least you have J to help you through this, but I know it's rough. Your hubby has flip-flopped back and forth so many times, maybe he will come around again. But such a roller-coaster for you!
Odd that you two were swingers before, and he's got another squeeze, but now he calls you a whore for loving another beside him. I've never understood the swinger mindset, but something is not getting through to him. I don't recall if he's in therapy, but it might help to start that, if he's not. Maybe it would help you, too.
I hope you and DH keep talking and trying to work it out. Has he ever come here to read this forum? Might also be helpful for him.
The world opens up... when you do.
Oh, oh, can't you see? Love is the drug for me. ~Bryan Ferry
"Love is that condition in which another person's happiness is essential to your own." ~Robert Heinlein