I agree with you and think there IS Polyphobia out there. Two things that burn my butt are:
1. People that automatically think Polygamous when you say Polyamory and IMMEDIATELY go down that path of "I think it's sick that a 65 year old man should have (insert number here) 14 year old girls as his wives." What has that got to do with Polyamory???
2. Who gave society the right to make any judgement at all about what consenting adults are doing in the privacy of their own homes??
Sorry, I'm getting worked up just typing this...I should probably avoid these topics.