Yeah, it is. Most folks only think of an additional lover in terms of cheating or swinging. I've had people say to me "Oh, you're swingers" when I mention we're poly. To that, I respond with "Nah, that's polyfuckery. Polyamory is about the relationships," which has seemed to work to clarify the matter enough that they cease assuming we just want to fuck around.
As for any rude comments, I just ask if they're serious about starting a fight. That seems to wake them up to the fact that they were seriously rude with the comments. It helps that they really don't want to fight with the veryvery large man (I think they're afraid I might eat them).