There are times I scratch my head over the difficulty folks have in sorting out the differences between poly and swinging. The best phrasing I've found to differentiate them is to refer to swinging as polyfuckery. That's cleared things up for lots of people. Fucking around is nowhere close to having multiple close relationships. Most folks know somebody who is always playing the field, picking up new sexual partners and never building relationships with them--that works as a basic description of swinging when applied to couples. Swinging is primarily about sexual variety and poly is about emotional variety.
It has to do with the strength of ties, too. When I was still involved with my recent ex, the discussions my wife and I were having about where to move in the future expanded to include my ex--she was part of the family and had a major stake in the decision. Were I just involved with her sexually, she wouldn't have been included in that discussion.