Sorry this had to happen to you - the guy was being a total jerk, and being drunk is absolutely no excuse - drunk people tend to do what's in their hearts - the alcohol just turns off their internal filters....
Yes, people misunderstand all the time what poly means - they apply their own standards to it and basically stop listening. Many assume that we are swingers, and that this is just a fancy name for it. One woman, who I had always felt attracted to, even after we explained the difference of what poly was and how it differed from swinging, pursued me (it turned out later) without her husband's knowledge, only for sex. That was messy, because I was better friends with her husband than me. That kind of ended the friendship with him, which I was VERY disappointed about. Since then I have learned that she has informed other friends of ours that she can confirm that I am definitely a swinger. Of course, they have never bothered actually mentioning is to me/us - but the gossip flew for a while.
Others mistrust the stated reason of why we are telling them, assuming that the reason is that one of us wants to have sex with one of them, or are in some ways interested in them. Why does that have to be the only reason?
We have had many of our long-established friends distance themselves from us because we talked with them about this.
"Listen, or your tongue will make you deaf." - Native American Proverb