There's a lot of overlap between the groups mentioned so far...but not everyone who's a member of one group is a member of any of the others, and obviously poly types might be a member of one of the groups...but they might not.
What do hippies, SCA-folk, sf fans, wiccans, BDSM types, and gamers have in common? A tolerance for those who are different, a willingness to just accept people for what they are and judge them solely on the merits of how they fit into that sub-group, not the larger world as a whole. "So, you're polyamorous? Whoa, that's cool." "Thy personal life is none of my concern, varlet! Now, have at thee!" "Really? I think Heinlein wrote some stuff about that; so did le Guin and Bradley. You ever read any of them?" "Yeah, okay. So, you going to roll to hit, or what?"
Part of this is because all of these groups endure judgements based on what people THINK they are, rather than what they ACTUALLY are. So, it's not surprising that many poly people gravitate towards groups that are accepting and non-judgemental.
But tolerant, accepting people are everywhere, if you look long enough.