Almost all people who have been hurt by affairs will tell you that what hurt worst is not the knowledge of the sex, but the lying, the covering of tracks, and the betrayal of trust. That's why these emotional affairs (EA) are every bit as bad, even worse, than physical affairs (PA). Although quite often an EA will become a PA when given the right cirmumstances.
It really sucks, and hurts a lot. I hope you will hold him accountable and really work on your marriage and only investigate polyamory when you guys have made major strides in your relationship. Keep us posted on what happens. We're all rooting for you guys.