If it helps, I had some of that feeling too at some point. I think it was how I reacted to the traditional model, and rejected it. I didn't want to be with other people, I didn't intend to pursue them, but knowing it was off the table made me feel locked in, and it gave me some anxiety, too.
I looked up a lot of things about open relations, talked with my husband a lot, tried to figure out what was for me and what was not, and after taking a look at swinging (by looking at swinging websites, etc) and figuring it wasn't for me, I finally learned about polyamory, and it felt like so much more of a match for me!
After my husband and I opened up our relationship, I didn't meet anyone I was interested in for something like two years. Yet these two years were so different! I felt freer, happier, more myself.
It's hard to know if you are the same way or not. First, because monogamy is the norm, and poly people do tend to question themselves and think there is something wrong with them, so thinking these things doesn't exclude the possibility of being poly. Yet it might be true that your problem is not there.
However I would recommend not thinking about it as being selfish. I think as long as you are as honest as you can be with your partners, that's what matters. A relationship in which you're unhappy for the sake of someone else isn't going to be a good relationship for either party. And all relationships should have a part of selfishness: find someone who makes you happy, who matches what you want and need, and vice-versa. That's the foundation of relationships. Of course, that doesn't mean you leave as soon as something annoying happens, but a problem arises and you discuss it and try to address it, sometimes breaking up (or not going further) is actually the best solution for everyone.
I guess for now, no matter whether you are poly or not, you should probably warn any potential partner that you think you would be more comfortable with an open relationship, and then figure out together what you feel works for you (and I mean the two of you).