Yeah, I totally agree - there are VERY few folks that I would be interested in having anything more than a conversation with. Having that whole "poly" thing restricts your available "pool", as it were. once you start weeding out those whose way of doing poly is utterly incompatible with yours, it gets to be slim pickings.
But on the flip side, I have found that compromising and "settling" ends up hitting back at you down the road, and causing drama and grief.
If you are getting frustrated (and it sounds like you are), take a look at your criteria, and explore if there is anything in there that you could relax as a requirement that might open up the pool of folks a little...