That's one of the complicated parts of communities that span typical social groups, isn't it? If you don't feel you fit into the "core" of the group, which in poly is typically nerdy and fringe (in my experience), then maybe you decide to never attend again. But, there are plenty of similar people who are also poly out there... enough so that one or two people who might become friends show up once a month. Did they show up on the same week? Probably not. If they only attend once, then they'll never meet, and so on... until all the potentially similar people missed each other. Then, they think there's no one like them in the community, but maybe only because they didn't stick around long enough in order to find each other.
Or look at it another way... Suppose you wanted to find friends at a cool grocery store because you want to meet people interested in good food (for your definition of good food). Would you only go once because you didn't find anyone similar that one time? Even if the chef at the demo that day put milk in the frittata eggs (or whatever thing would be offensive to your food sensibilities)?