Personally, I think the antipathy between "poly" and "swinging" is kind of... I dunno the word. I want to say collectivist, but it's not what I mean.
When I first began exploring myself, I connected with a poly group in my soon-to-be area. They had a heavy emphasis on children and families, and seemed defensive that I'd consider sex to be a big part of my relationships.
It didn't lead me to feel very comfortable and welcomed.
And with the variety of relationships that exist in the poly world, I think classifying swinging versus poly sort of shuts the door. My wife once thought she'd engage in casual sex with other people, but she pretty soon found that it's emotional.
My push towards poly was spurred by the desire to break from the "burden of one". My wife was my best friend, financial partner, lover, emotional support, and all of that. There are some things that two unique people can't provide for each other, and opening yourself to relationships as they work for the people involved is everything poly is about to me.
If I had a sex partner that I trusted intimately, but didn't contribute to my finances and share my hobbies, I'd not consider them any "different" than my partners who shared their sex life and companionship with me, but didn't fill the other roles of my life. Each aspect of my relationships serve a purpose based on how we interact. I'd still call it poly if the person or persons I'm involved with were a part of my sex life but not really present elsewhere since that's the relationship I entered into.