If you're not into the idea of having a threesome- tell him so. He will just need to put on his big boy pants and get over it.
Honestly, it sort of feels like he's telling you he's been fucking around and expecting to get rewarded for it. You may learn to accept and deal with the issues in your relationship, you may even find that you prefer both of you to be polyamorous, but that does not mean that you have to fulfill his every fantasy.
On that note, since you feel that you may want to be with another woman and would enjoy one-on-one sex but not a threesome, you can share that with him. I mean, if everything's going to come out in the open, might as well share your thoughts as well.
I'm not saying he has to be the villain here (though he would be if he was my husband) but what I am saying is that he should also understand your feelings and be more open to working through your relationships issues before peering up others skirts as an alternative (so to speak).