Mono and I talked about your situation last night. We disagreed on several things but here goes with my opinion any ways.
The man said he felt as if he were cheating right? I would suggest that he guessed when it came to understanding your point about it all in terms of it being about love, not fucking each others wives. It probably seemed like a great idea at the time as he didn't think he would react. He probably thought it would be fine to give his wife to you and would be fine to take your wife in return. He missed your point and, not only that, he got all excited about the sexual act, had sex with your wife and then realized what he would lose which is his wife. He got emotional, which was your point. It was about emotions for you. It turned out that he actually cared about that enough to say stop!
I agree, he's an asshole, but give the poor man some credit, he saw where he went wrong and is working on it. They both are.
How many times have we said on here that poly relationships challenge EVERYTHING! They really do and I really hope that at some point your anger subsides and you are able to empathize with them. For yourselves more than anything.
Perhaps you can use this time to better educate yourselves on poly and all it entails before proceeding with another crush/love interest.
Too bad you didn't get laid. I gotta say, that really bugged!!