I see what you mean. I see my husband as more "open" than "poly". He is the same in other aspects, for instance when I turned vegetarian he didn't, but he had to adapt to the fact I wasn't eating animals anymore (just like I had to "adapt" to the fact that he still would, so while I wasn't eating meat, it would still be around the house).
To me, it's important in making relationships work. How much you're willing to compromise when your partner is different. Many vegetarian/non-vegetarian couples just wouldn't work because one partner would try to change the other, for instance, or just say it shouldn't affect them what the other does, and leaving when it does.
But communication and strong ties and complicity help a couple overcome their differences, sometimes one partner has to compromise more, sometimes it's more even. When one partner compromises more, it's important that the other recognises it. At the same time, it's important that the first one realises that while the other isn't compromising as much, they still are.
About the idea that one person might be poly on the emotional stage but without sex, I do believe it's possible. Actually, the way I see it, polyamory focuses on love and feelings and relationships, and swinging focuses on sex. While they overlap, sex without feeling would be swinging and not polyamory, and feelings without sex would be polyamory and not swinging.
Sex with a friend could be either/both, I think.
So I think that's the main difference, a difference of focus, and that by definition polyamory focuses more on feelings than on sex. That doesn't mean there can't be both, since there usually IS sex as well, but that does mean, in my opinion, that there can be feelings without want/need sex and that would still qualify as polyamory.
I guess that would mean you are either not a very sexual person, or exclusive for lovemaking but not love, or perhaps in some cases you might think your feelings are wrong and prefer thinking of the other person while having sex with your "real" partner...
Either way, in my opinion that still sounds like polyamory, while cheating or swinging without feelings doesn't "count" as polyamory to me (although that's not monogamy either).