I'm new to this board too, but I've been on another poly related board for awhile which has helped me.
Your situation sounds similar to my own and to a lot of others' here; where what began as a bedroom fantasy ended up spilling out into real life.
Based (only) on what you are saying, it seems that your husband is not being straightforward or honest with you about his attempts to pursue other relationships at all. Whether this is based on a larger problem of communication within the relationship or not, I don't know.
At any rate, what I do know is this type of secretive behavior is destructive, no matter what type of relationship you are in. If he wants to stay with you (i.e. not destroy the relationship) he should understand this. If he feels he cannot be up front with you because you cannot handle the news without freaking out on him, then both of you should check out books and media on nonviolent / nonconfrontational communication.
It sounds like you need time to figure things out. Having a long-term partner request a game-changer like this, and especially doing this in a non-constructive way, is a lot to chew on.
For your part, it is OK to not want to be in a poly relationship. There's nothing wrong with you if you're not OK with it. Some people in these situations do some deep self-analysis and realize they can make poly work; other people do the same thing and realize they cannot make it work. You need to figure out what truly, honestly, works for you, and then take that to your husband. It may be beneficial for your husband to stop seeking poly activities so you have the time and space to mull things over.