I haven't read what the others have said, sorry about that, I don't have much time and wanted to stress that he thinks he is cheating with your wife, he IS cheating on someone and this whole thing has some less than ethical ramifications for all of you. Not poly at all to me. Poly is about living with integrity, trust, honesty, open communication and empathy.... cheating does not fill any of those things.
If I were you and your wife I would chalk it up to experience (yay, you might not get jealous), and move on. I also would tell the guy, if I were your wife that while the petting was fun it was not right and that I never want it to happen again without full consent from his partner, just as I did. Then I would go find a poly person and skip the cheaters.
It sounds like taking a break to get a grip on what your ethics are in all this might be a good idea. I suggest looking at the lessons learned
thread and the foundations
thread to see if there is anything that might be helpful in gaining some perspective on where you are going in this.