Perhaps you are instead angry at the miscommunication that occurs due to incorrect use of the term? Words represent specific things, actions, and concepts. When one uses a term incorrectly, such as polyamory, it leads to thoughts and action based on the assumption that an idea is being communicated accurately. I can see being annoyed by this with the situations you described.
That said, I often find myself annoyed when inaccurate communication of any kind occurs - not just that which concerns the use of the term polyamory.
In any case, there are so many ways to practice polyamory, it probably would behoove all new acquaintances to take the time to spell out his or her particular version.