Tomte, I think if you want to salvage your friendship with this guy, talking to him is inevitable. You mentioned feeling like he was disrespectful to you, especially since he has no idea of your poly leanings, and I think he should be told just that -what he did felt like disrespect, which is an entirely different feeling from the jealousy he might expect you are feeling. Telling him is a way of demanding and regaining that respect, I think. It could save the friendship.
I'm a little uncertain exactly what you are feeling as far as your girlfriend's part in this, but everyone has good advice for you. I hope you find clarity.