Have all of Adam's polysexual activities been in front of you, or has he slept with anyone on his own, without you present? Sometimes threesomes and voyeurism can be a good first step into polyamory, but still keep us safe, because we are still involved in the situation... therefore, feel more in control.
There are many different kinds of poly and polyamory doesn't suit everyone. It doesn't make you selfish, or less of a good person, if it's not for you. That being said, it's not to say that with some work, it couldn't be for you!
I entered my current relationship two years ago with a polyamorous mindset - love as many as we want to. Over time, that became more of a polysexual relationship, with a hierarchical framework... myself, my GF and her husband are in a V, with her as the hinge (i.e. not a threesome, or triad)... seeking other partners for sex and friendship. We are exploring and getting a bit more comfortable with the idea of loving other partners... and of course, all recognise that this can happen.
You're not selfish or unusual in your thoughts. What you are going through is a completely normal reaction to a fear you haven't had a chance to conquer yet. Look at it that way.
The proof is also in the pudding. It can be way, way scarier to think about Adam falling in love with someone else than to actually live through it. If, in six months from now, he's in love with someone else, but still being wonderful to you and just as committed, you might be able to see that... wow... actually.... this isn't so bad for me afterall. On the other hand, you might find that his love fizzles for the other girl, or... that his love fizzles for you. Unlikely that it will fizzle for you - but it's always better to live through experience that to run from experience through fear.
Watching your partner fall in love with someone else, especially the first time, can be really hard. Not only because it's hard for you emotionally - but because it's the first time you both have to navigate such a thing. He has to learn how to balance, how to discover whether he truly can love more than one, how to be EFFECTIVELY sensitive and reassuring (it's not as simple as just saying "I love you")... and you have to do the individual soul searching and emotional work.
As for the fact that you like this girl.... hehehe.... well, for me that doesn't always help. My GF fell in love with a guy online and he was so wonderful... that actually became even more of a struggle for me, because she would go on and on about how perfect he was (even gave him a nickname of Perfect [Name]). Eventually, after you've seen your partner get attached to people you don't like.... you find yourself doing a U turn and finding the nice people easier again.
Compersion is not always the only emotion in poly... in fact, it often goes hand in hand with those negative emotions we feel. Feeling compersion about watching him have sex, or hearing about it, is actually a kink. Voyeurism and/or cuckolding are the terms for getting enjoyment out of watching a lover with someone else. If you feel an emotional happiness when watching that, then this is more like compersion. Don't expect yourself to feel full of love and joy right now.... instead, expect yourself to feel fear. It's completely ok. You will be able to work through it and in the process, find out what works for you.
For me, poly is about trial and error, self-discovery, adaptation and growth. Jealousy doesn't necessarily fade with time... though it can. Someone on here once told me that being poly is like being at University - you can study something for years, but still feel anxious about an exam you've never taken before. If you watch your boyfriend go through various other loves, you will probably find it easier and easier to deal with. If you see him stay with this girl and still be devoted to you, you will probably find it easier. Give yourself time and ask for his patience.
A few good things to read...
Google: More Than Two; it's a useful website.
This article, posted by Gala Girl, is brilliant:
Sternberg's theory of love has helped me to understand that just because my GF falls in love with someone else, it doesn't mean that it's immediately the same love that she feels for me... and it definitely doesn't mean she loves them more...
Tennov's theory on Limerence could also be very helpful... it describes that new, infatuated, 'crush' love that we tend to feel when we meet a new person... this theory has helped me laugh at NRE and learn not to panic when I see my girlfriend getting rose-tinted over a new person: