That has been my experience generally as well. My local self-identifying poly community is mostly lower middle class white people. There are some working class people and very few upper middle class people. One could assume that wealth gives one the ability to do what one wants and not worry about social impacts of being poly, for example. But it seems to have the opposite - those people with the most to lose, wealthier folks, and those with the least margin for error to survive adopt a similar strategy of not being marginalized. Of course, I have absolutely no proof of this - just my own observations.
There are many African-American, Asian and Latino people active in my local community. However, I do live in a majority black city and so, the relatively imbalance is certainly present.
I also agree that if one is already part of something not mainstream, poly (and kink for many people as well) becomes more possible, more 'thinkable'. I also believe this is partly why there is heavy overlap of poly with folks who are also pagan, or geeky, or both.