I've seen a few people post on other forums who are vehemently opposed to the idea of swinging and disliking anyone willing to participate in sex-for-sex's-sake. But, I've never met people like that. I've met plenty of people who think non-monogamy is a continuum between poly and swinging, where most people fall somewhere in between. I've heard others with the opinion that poly and swinging are unrelated and you can be/do/have both kind of relationships.
On the east coast, my experience is that most people don't care so much. My understanding is opinions (and definitions) can be very different in other areas.