Yep, it's usually envy I tend to feel. Hubby works out of town and when he takes advantage of non-monogamy, he does so when on the road. So I wouldn't be getting it anyway, but I can't help but feel some envy for the person who is. I don't let it stand in my way and it goes away easily with a bit of rational thought, but the first reaction is that small pang of envy.
I was well loved and attended as a child, so I have pretty high self-esteem. I don't tend to get jealous and insecure, and I usually recognize it as such pretty quickly and nip it in the bud. Usually that's just a matter of declaring that I feel such, and then deciding to manage it.