You sound like my last girlfriend.
Sadly, she and I had to break up because her hubby couldn't get over the fact that she and I were in love.
I barely ever see her anymore, and while it hurts, I shove that hurt down even deeper. I actually WORK with her hubby. (Odd right?) He is my supervisor at my job. Oddly enough, he STILL got me this job even AFTER knowing about his wife and I.
We all started out as swingers also. My wife and I and her and her hubby. We were all swingers, but my wife and I only played with them. They would OCCASIONALLY play with others. Mainly AFTER my wife and I introduced them to the local swingers club. Boy was THAT a mistake.
Anyway, we would meet up with them every weekend....This went on for 3 years! Eventually, after about 2 years, I realized that I actually CARED for his wife...... I confronted her, and asked her if she felt the same. She confirmed that she did, but would never leave her hubby. I told her I didn't WANT her to leave him, and I wasn't going to leave my wife. So my wife and I started talking to him about polyamory. All he could clean out his ears to hear was "polygamy".
I tried and tried to explain it to him, but he got stuck on "poly", and made up the rest himself after that part of the word.
So, here I am now. My wife and I are currently dating a wonderful single woman. We are happy. I DO have times of longing, but I get over them.
I have to ask, did you at least explain the problem with the new man? Or did you simply never speak to him again leaving him wondering what he did to make you mad?