I have a somewhat similar reaction -- when I had a poly lover I was fine with her seeing other men and women, but I couldn't allow myself the same freedom without feeling uncomfortable.
In my case I've retreated from poly, at least for now. I don't know your full interpersonal situation nor your needs, of course. If you're getting what you want and need from your lover, then that's good!
Do you feel an internal pressure to look for other men? Is your subconscious telling you that things are unsatisfying, or is it purring and telling you that things feel really good as they are?
Is it possible that self-awareness and time will let you relax into loving other men? All this stuff takes considerable understanding of oneself, I believe. Such understanding can be slow in coming when one is developing and changing. It sounds like you've gone through a period of growth recently.
If you can, take your time and give yourself room. I rush things, myself, and damage both my own feelings and the feelings of others through my impatience. Therefore I recommend patience even though I have great trouble with it myself...