I guess that I feel that swinging is one form of responsible non-monogamy if it is done well. Like any other it can be done badly, and there are predators out there who use the label as a way to deal with their own selfish, manipulative needs.
While I haven't actually tried swinging, I did go through a phase on casual sex outside of my relationship (with the consent of my partner). Often these were one-time deals with friends where we were both "in need", but once it was a session of sport sex with a woman I had chatted to on the Internet for a long while. I think that if I had really decided it was for me, then swinging would have been a very legitimate lifestyle choice for me.
But I decided it wasn't, and I have thought long and hard since then about why not. I have good friends who are, or have been swingers, and I enjoy their company and can definitely appreciate their ability to swing.
Good sex, for me, has such a strong mental component. Without that it's just "Insert Tab A into Slot B" - enjoyable for what it is, but missing so much of what I want out of sex. During the times I was having sex with folks for whom I didn't have that mental bond, it almost felt like I was apart from myself and looking at myself from outside my body. I felt like I was watching me in a porn video.
One friend of mine said that I was a very strong empath, in that I feed off the energy of others. I think that this reflects very well my attitude towards sex - if I don't have that mental connection with someone, or that person can't feed back to me the sexual energies that are going on inside them, and I don't feel that what I am giving in terms of feedback is being received, then the experience doesn't seem complete to me. Doing it just for myself may get me off, but it feels like McDonalds where what I really enjoy is a degustation from a talented chef.
Part of me really wishes I could enjoy casual sex more, for the varied experiences that it would offer, but I just don't think that I am wired that way, in the same way that I know that I am not wired for monogamy. But I still see both as highly legitimate options for relationships.
"Listen, or your tongue will make you deaf." - Native American Proverb