I totally feel your situation here. I've been in the same boat, and now that I'm in an officially-declared poly relationship, the soul-searching and work really hit me. It takes some time, some patience (with yourself above all parties involved, but all-around patience is necessary), and dedication, and love...there is a lot to be done.
I, too, indulged in the "for play" conversations about who we'd do, keeping the relationship open for testing purposes, etc. and I, too felt seriously threatened by her interest in finding other mates. I, too felt serious insecurities and self-doubt spring up. It helped me a lot to talk about how I was feeling with my wife and make sure she knows how important our communication, clarity, and her support are to me. Without her constant reminders of how much she loves me and how she's really not "going anywhere" without me (leaving me), I would be pretty lost, I will admit.
Unfortunately, many poly people get comfortable with the idea of an open relationship and don't realise that when monogamous people live in a monogamous relationship for long enough, they start to expect that it will always be that way. It seems to me that even though we say "yeah, we're cool with it," a proper serious conversation needs to be a standard thing that all poly people do when introducing the topic. To make sure that we really *understand* that they are *actually* poly and *always* will be. And they need to remind us from time to time, too, especially if there isn't another partner in the picture for a long enough stretch of time. I got caught by this, and it seems to be the major falling point in mono/poly relationships. I'm sorry it seems that you got hit by this, too...but it's almost unavoidable. All you can do about it now is catch up, and we're here to help you with that.
The important part to remember is to talk, to breathe, and to relax. It's not the end of the world, even though it may feel like it sometimes. Open communication is absolutely necessary, as you already know, so talking about your feelings, owning them, using "I" language, and being very careful and clear in explaining how you're feeling or thinking to your partner will absolutely save the relationship from the stress that will build due to any perceived threat. Just remember that oftentimes the path one takes to avoid something often brings them to meet it instead. Act out of love, not fear, and you will make it through this just fine.
Have a serious talk with her about it, about her being poly, about how you feel, and don't feel bad about asking for some more time to get used to things. Ask her to take her new relationship a bit slow, so you have time to grow into it and adjust...unless she's already jumped in (in which case, the communication is not just important, it is now mandatory).
You're not alone. This can be a support group of sorts, but there are also area poly groups in various places around the world. Try and find one near you, they do exist.
I agree with Breathesgirl. Xeromag is fantastic. There are a lot of good articles and a lot of good links. Polycat (it's a blog) led me to a few great articles too, though I just lost the bookmark for it, so Google is your friend there.
Remember that while you might be hard-wired for monogamy, it's more likely that Mother Culture has simply taught you that monogamy is the only way to be, and that is most likely the cause of the majority of your discomfort. Screw society, follow what actually feels right. Don't think too much, because that's when societal influence comes into play. Just feel. Be in the moment, be present in your body, and don't get lost in self-destructive thought cycles.
Also, remember, you don't have to be the "mono who changed his mind," so to speak. You don't have to become poly, you just have to come to an understanding that she has multiple parts of her that she needs to fulfill with different people, and you fulfill a very important part of her. Asking her to be fulfilled only by you would be asking her to change who she is, so the only stumbling point should really be accepting that she is fulfilled by you, that you don't mean any less to her, and that she will probably even feel her love for you more strongly due to her feeling love for another as well. My relationship with my wife got much stronger once I relaxed to the idea of her dating other people, because she was able to energize herself rather than holding back and only loving me. Love in poly people seems to defy any laws of physics or science: the more love a person expresses, the more love they generate, and the more love they have to give. It kind of just grows like that. It's pretty cool to be in on such a fountain of love / love generator. We're lucky to be with people who can generate love like that. That means there's that much more for us. Not less, but more.
Welcome to the forums. Welcome to polyamory. If you need somebody to talk to on a personal level, feel free to PM me. I'm most certainly not going to be the only member here who will offer that.