Yes, it definitely sounds like the major issue here was sex without an established relationship--or at the very least, a friendship. You seem fine about the bf, the gf, and the couple. (After all, you know them and aren't threatened by them.)
But this one guy and how he and your wife went about things was beyond your comfort zone. It sounds like eventually you let your wife know and the fact she broke it off is commendable. Hopefully, lesson learned about that approach and how it's a bad one.
Those lingering feelings are left over and it's something to work out...with time.
This is not the first time I've heard of a couple going poly and the wife having a way easier time finding secondaries....you seem to have to search hard, while she gets asked out left and right. That's not unusual for poly MF couples, sorry to say.
Concentrate on your needs and wants and try to keep a trusting line of open communication with your wife open.