This hasn't happened to me specifically because I am poly - because not too many people know that. I have has somewhat similar experiences, however, when people know that I am bisexual. Just because I can be attracted to both men and women doesn't mean that I am attracted to YOU - Jeesh! (This harkens to straight guy fears that all gay men want them - really? Are all the straight girls falling at your feet as well? You seriously think that you are "all that"?)
Apparently poly and/or bi means "sex fiend" to many men. Hubs suggests stomping on their instep before kneeing them in the groin - YMMV.
Me: poly bi female, in an "open-but-not-looking" Vee-plus with -
MrS: hetero polyflexible male, live-in husband (23+ yrs)
Dude: hetero poly male, live-in boyfriend (4+ yrs) and MrS's best friend
Lotus: poly bi married female, "it's complicated" relationships with Dude/JaneQ/MrS (1+ years)
+ "others" = FBs, FWBs, lover-friends, platonic G/BFs, boytoys, etc.
My poly blogs here:
The Journey of JaneQSmythe
The Notebook of JaneQSmythe