See I'm late coming to this thread so if I repeat anything others have brought up - sorry pal.
I think it's normal to form 'associations' with certain terms based on what we know and/or what we've experienced. At least you have been honest with yourself about your own association.
But it's words. Nothing more. The trick is to break the association - because you might be avoiding some very wonderful people purely out of word association.
Like polys, monos, or humans in any culture (race etc), there's good and bad everywhere. As someone who has dabbled some in what some would call 'swinging' in the past, I can only tell you that the spectrum is as broad as any other. It's why I have raised eyebrows here at times at those that wanted an unlimited condemnation of anyone with any association. As you probably heard me mention, it's surprising the number of labeled 'swingers' that really are looking for poly partners and aren't aware of the lovestyle at all.
So I'd suggest if you want to continue to paint with the broad brush you just accept that you could be painting some wonderful, loving people out of your life. It's one choice - nothing more.