For starters, stop telling stupid people that you're poly.
I don't tell anyone about my "lifestyle" unless I know they'll understand it or that they love me & accept me for who I am. So perhaps you're telling the wrong people or perhaps you need to read some definitions of poly so you can come up with a concise & meaningful explanation that explains it. To me poly is about loving more than one person at a time, it's as simple as that. And yes, sex is often involved but that's not all of it by a long shot. Also keep in mind that poly takes many forms. Also keep in mind that most people are average or less than average, which means that have limited intelligence & experiences to help them understand such things. If you think they're smart enough have them read Sex at Dawn or something on being poly. Monogamy has been so ingrained in us even with the glaringly apparent divorce & infidelity rates.
People often boil everything down to sex. I notice this with homosexuality debates. Even the most sexual people aren't having sex all that much if you think in terms of a 24 hour day. We do many other things in those 24 hours, many things other than sex & yet everyone is focused on the "yucky" gay sex part. They don't think of the many long-lasting & loving relationships of many gay individuals & just focus on the sex....totally predictable & totally lame.
So STOP telling dumb people you're poly or be prepared for stupid comments & come up with a good comeback...London had a great one ;-)