"What to say to people who assume poly = group sex?"
Um, this might work: "No, it's not."
Seriously, though - I rarely use the terms polyamorous or non-monogamous. I just say that "we're not exclusive." Talking about exclusivity, or not being exclusive sounds more like it's about dating rather than fucking, I think.