If you want to avoid "labels" and can't agree on TERMinology (first four letters capitalized to emphasize the root word) with respect to the noun or adjective (polyamory, polyamorous), then instead of defining the relationship by what it IS or IS NOT, you might try defining a relationship by what you DO instead, of which the broadest expression would be "see other people".
"My partner and I both agreed that we can see other people" goes down much smoother than "My partner and I ARE polyamorous" or "My partner and I ARE swingers" or "My partner and I ARE NOT monogamous".
Then, you can elaborate on it if someone asks "So what does that mean?" if you want. But, I think it helps to put things in terms of action-words and -phrases even if it means being very general and encompassing at the outset. It's better to start out general and work your way toward being specific than it is to start out being very specific and having to expand your list of exceptions every time someone makes a counterpoint.
(and this is the general-you, not "you" as in a specific individual on this forum. Gaud. I should put that in my signature. I have enough Facebook friends for now...)