I tend to see "poly" as living a lifestyle where a person may desire loving many people with knowledge and consent of all people. So in the case where the partners do not love the people they are intimate with, I would consider them swingers. If a person is single, I consider them poly if they are willing to be involved in a poly relationship (just like someone can be a gay virgin).
Ultimately, there are two reasons for labels. One is to help identify others in a consistent manner (like for sociological research) and the others is to identify yourself as how you see yourself. So if anyone self-labels as anything, I go with that since they are trying to explain something about themselves. However, if we are talking statistics, I like a consistent label.