I do poly. I don't swing.
There are folks who are like me in that regard. There are also swingers who don't do poly. To claim that the two are the same thing directly contradicts the experience of the people actually living their lives. That some people do both doesn't change that.
To claim that a sex-only relationship qualifies as poly because it's a relationship also doesn't follow. That standard would mean that any relationship--with your doctor, your banker, your cousin, your mother-in-law--is the same as any other, so everybody is poly because of it.
Unless you're trying to claim that it is simply sex that defines a poly tie. That also doesn't match the experiences of many people, so it can't really be said to be accurate. All the Johns picking up hookers this evening are not poly because they're having sex with prostitutes. The playas in the clubs picking up one night stands are not poly just because they're having sex. And so on.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.