Thank you everyone for your replies.
I'm not totally sure where my guilt comes from...
It partially is from my raising-- I went to Catholic school until I came to college, and though I never agreed with a lot of their teachings, it seemed to press the monogamy mode into my brain. I know many people who are poly, but I didn't think I would ever be with anyone besides my partner.
Another part of the guilt may be fear. I AM afraid that I won't be able to control emotional feelings from developing for others. I know that my partner has become more secure in recent years, but early in our relationship, he did have some distrust about me being emotionally involved with others.
What if I did develop emotions? If I don't want that, can it still happen?
As I said before, I've never had sex with anyone except my partner, so there's just a lot of unknown for me.