I can see where there were really mismatched expectations. I imagine he took the fact that you had said you were fine at other times to mean he could do as he liked, because well...some people don't really talk or think. Have there been other non poly things that this has happened with, you thinking you had an understanding but later finding out you weren't on the same page?
Me, I'd be PISSED off, but I am very clear about what I want or need ahead of time so if my husband did that to me, it would be a clear betrayal of our agreements. Don't think you need to be pissed off, but would understand if you were - but who would blame you if you were hurt or confused about why he didn't mention ahead of time he was going to look for a sexual partner when it had been on his mind. Why wouldn't you be scared? It can be scary when you are surprised by a partners behavior. Sounds like you may be a bit like me in some respects, you let your partner know everything that goes through your head, wants, desires fantasies, hopes and dreams, etc, so expect and hope that they will give the same back to you. It can be hard to understand when they don't do that. However, it's a good sign he was able to answer all your questions about what he did, it doesn't sound like he was avoidant or defensive, so I imagine he feels he didn't break any agreements, and wouldn't make it the biggest deal in the world (unless he wasn't planning on mentioning it if I hadn't brought it up - then I would be exiting that relationship just for my emotional well being - somebody with expectations so far off from mine that didn't feel it was important to see if we were on the same page would just mean years of struggle I feel).
The important thing now is probably to to clarify what you want and need or would prefer, be it a heads up that he desires to seek partners without a pre-talk (or to be honest that you are not feeling OK with that), discussion about if you are both free to seek partners separately and how you each feel about that, including clarifying safe sex agreements. It's also a good time to see if your wants and desires are still compatible, and smart to be open to the possibility that you may have different ideas about how you want to go forward (though I hope that isn't the case). If there are things that you don't want to deal with (like random hookups when your partner is out of town) it's the time to be clear about your boundaries, the ones that are definite and the ones that might be flexible.
I really don't read that you are jealous, just that you're a potentially great partner to have in a poly relationship, with a partner that isn't forthcoming with his own wants and desires. It may turn out he is operating under less than ideal circumstances, and hopes you'll just put up with whatever he does instead of having to have the maturity required to sit down and let you know where his head is at and make sure both of you are doing OK. I'd guess that if you can't get him to sit down and talk clearly about this stuff, that he probably isn't a viable candidate for a happy healthy poly relationship at this point in his life.
Happiness will never come to those who fail to appreciate what they already have.