Ok - think we'll agree to disagree on this one. And I'll be shocked if you don't get flayed by some of our resident defenders of labels,stereotypes etc. <grin>
The "divide" you speak of is one we choose to build - or not. You do. I don't think everyone else may. That's ok.
My point was that (if we even agreed) that "swinging" was "all about sex" as you say, that there's a good chance that that's what the participants "think" it is - for lack of sufficient self analysis. And through that gateway they MAY discover more. More about their own true nature & desires. That the "sex" - although maybe some small part - was NOT what that "pull" really was at all ! At the root of it was a desire for some deeper connection they felt they needed.
But I think your analogy i.e. golf, comes up short too. A better one might be between swimming and a triathlon