On a side, rather random note, every swingers club I've been to that allows solo females also allows solo males. Sure, the cost is usually a lot more, but that is because of supply and demand not because swingers as a whole disapprove of solo men.
I have heard of certain groups that host parties that will allow solo women but not men, but they seem to be few and far between. Single women in the swingers scene tend to invite drama and bad behavior. At least the men only seem to invite bad behavior without the drama.
As for the poly side of things... I don't know what to tell you. I personally am a person who likes to surround myself with those I love as often as possible. Very rarely do I feel the need for completely alone time. So, I wouldn't be interested in a relationship with someone who needed a lot of time completely to him/herself. But that's just me.
I also draw a very solid line between my NSA/swinging experiences and my poly ones. My past relationship with someone I met through swinging was awkward and almost entirely unpleasant. I won't let those two aspects of my life mingle that much ever again. So, if I met you at a swingers event or some website geared towards NSA/FWB, I wouldn't have any problem hooking up (assuming we were interested in each other lol). If I met you on a dating site or one geared towards poly, I wouldn't be interested in anything but friendship on the basis that we want vastly different things in the long term. I don't know how many of the poly people you've met feel that way, but that could be one reason they haven't been interested.