I think it has to do, in large part, with not having any real agenda to push. Concerns about backlash from employers and such is part, yes. I just think the major part is that we don't have anything we want to push.
It's just like being Pagan. Aside from being Pagan, what do Pagans have in common that they might want to push for? Pagans come in every political stripe and have highly varied beliefs, as one would expect from the myriad religions that are grouped under the Pagan umbrella.
Poly folk likewise have different religious backgrounds and beliefs and different political philosophies. What would we hope to achieve by organizing?