You're not alone
And nearly identical situations are responsible for breaking many relationships. After reading what he said, and the words used seeming to indicate he isn't someone who can live poly, you need to talk with right away. Like others have mentioned, you don't know the details of what led up to that text. Waiting to talk is only going to make you more prone to lashing out and blowing other seemingly small things into the bigger issue that can eat you up inside if you let it.
When you blow something out of proportion, or just snap at him for something trivial without coming forward later to say, the reason I snapped at you had nothing to do with whatever it was you snapped at him, it can be very damaging to relationships between people who genuinely care about each other. It causes a lot of confusion that doesn't get cleared up until the truth comes out.
Like others have said, don't worry about what anybody else will think if it doesn't work out. But an important thing to realize while you and your husband are trying to sort everything out is the path and pitfalls you two have found yourself in are common mistakes that people who genuinely love each other, yet still tend to fall in as you are learning how to love with more freedom that society typical allows themselves to.
You've learned that it's impossible to have just the fun parts of relationships without training yourself to have more casual, more like sex through booths with holes in the wall relationships. Some people can do it, but feelings can't always be completely refrained. Hopefully he has learned, or is in the process of learning how easy it is to have unrealistic beliefs about other lovers when the two of them do not have to deal with reality. Because if he doesn't realize that, he will turn into a serial monogamist and be disappointed and the relationship will fail because no relationship is like a vacation.
But once you learn those things you can allow each other to have "vacation Loves" but you will be wise enough to know how and why it isn't real. It can be experienced and enjoyed, but thinking it would be better off with this primary without realizing it is not an "in reality viewpoint" will likely do damage when reality comes crashing in.
Yes, it was unrealistic to restrict emotions with other sex partners, but yes he should have fully disclosed his feelings to his wife. Yes you shouldn't have snooped but if the two of you can create an environment that doesn't set you up for a tendency to omit details from your talks (like feeling feelings that are not allowed) he would probably fully disclose and then you wouldn't be suspicious and snoop behave there wouldn't be any peculiar behaviors that didn't jive with his talk.
Humans are much more intuitive then we are capable of fully understanding. The two of you are in some emotionally hard to deal with times, but they are lessons that if dealt with by doing the next right thing, you can be the perfect poly couple everyone thought you'd be, just don't worry about doing that though. If it's meant to be it will be, just realize if you both want it and can be humble and forgiving while learning these lessons (which is NOT easy as you well know) You Can Do It
If you want it, do it.
If you don't then don't, just realize that most people have run into the same problems, and at this point they seem to be such honest mistakes that you can still be the wise and successful poly couple that became that way without destroying the relationships while you learned your wisdom.
Good Luck and I am sorry you two had to experience this, but the pain will pass