Oh, I'm so sorry to hear that! At least you have J to help you through this, but I know it's rough. Your hubby has flip-flopped back and forth so many times, maybe he will come around again. But such a roller-coaster for you!
Odd that you two were swingers before, and he's got another squeeze, but now he calls you a whore for loving another beside him. I've never understood the swinger mindset, but something is not getting through to him. I don't recall if he's in therapy, but it might help to start that, if he's not. Maybe it would help you, too.
I hope you and DH keep talking and trying to work it out. Has he ever come here to read this forum? Might also be helpful for him.