The swinger says sex is not love, and that sex outside of a loving relationship cannot harm the loving relationship. The polyamorist says one can have more than one love, and that love for one does not detract from love for another. Some can hold both views simultaneously. Whichever view is held, both fall outside of what society deems "morally acceptable." Both views are judged by society to be "wrong."
Some swingers argue that love and swinging should never mix. Some polyamorists argue that sex should only occur within the context of a loving relationship. These are also, of course, judgments. The fact that you are a swinger or a polyamorist does not prevent you from making them. We all make them. How you choose to let those judgments rule the way you treat and view others is entirely up to you.