Oh many of us here know exactly what you're talking about!
I've read some say that the Polyamory community is sorta where the gay rights movement was thirty or so years ago----you get some limited acceptance here and there from the more liberal-minded, and then everyone else acts like it's some gross oddity.
I don't think I've ever seen the word Polyphobia out there...but I am sure it exists. But I think it says more about those people than it says about you and how you live.
For instance, while most monogamous gays have been accepting of us, there's others that are giving us many of those snide remarks you've been getting. But get this...after some probing into their thoughts, I've come to realize that most of the guys who say these things are actually jealous
And the more and more I look into it, the more I think polyphobia is built out of a jealousy. I can't count the number of times I've heard "I can't hardly get even one man and you got two!!".
Society has forced the notion of monogamy, even though there's mounds of evidence out there that says it really isn't natural. So some folks who probably deep down inside, in the back of their minds, would love to have multiple partners....lash out that they feel they just can't!
So anyways...that's my theory. Best I can do is explain, that they've really nothing to be jealous of....having multiple partners is not easier than having one. But of course, they'll deny up and down any jealousy!