Yeah, a word is actually only as worth as much as the amount of agreement that can be had about its definition (in any one given conversation). And I think I, too, on those rare occasions where I come out "to one of them" (e.g. to a new shrink I might be seeing), I don't bother saying "polyamory" either. What's the point? I just tell them that Snowbunny is married to Brother-Husband but also lives with me as if I were a second husband -- and both of us guys know about that and are okay with it. I'd have to explain all that whether I said "polyamory" or not, so I might as well not complicate matters by introducing some "kewl new word" to the conversation.
Re (from OP
"I have a hard time accepting that many full-blown swingers describe themselves as polyamorous."
As far as I can tell, that sentence is the actual original topic in this thread. But we've also discussed in this thread a couple of ways a person can be "swing" and "poly" at the same time. That admixture of contrasting ways of relating makes things messy, but it's part of the reality we all live in. How we deal with that reality is perhaps tangential to the original topic.