I think it's a matter of detaching sex from feelings.
If sex is tied to feelings, it might seem weird for you to have one without the other.
But taking sex as a pleasurable activity without the emotional attachment, it is similar to practising a sport together, or any such activity. You do have respect for your partner who is playing with you, but you don't lose yourself in it or give yourself in the act. You do it for the act itself, and not as something to grow closer to the person.
I think casual sex isn't for me, but I don't think it's wrong. It's possible that you equate it too often with men abusing women, but I think that happens more often with non-swingers who have one-night stands with women without telling them it's just for one night. With swinging everybody knows the facts from the get-go.