Welcome to our forum.
Well first of all, I don't think you're obligated to label yourself as anything, especially as so many of the non-monogamous terms are so relatively new in the language, and often subject to private interpretation. But, I see your point about wanting to set your boyfriend's mind at ease.
It is odd to me to hear the idea that monogamy and polyamory (mono and poly) are the only two relationship styles, and that everyone is either one or the other. The word "polyamory" has had quite a little history so far, and "responsible non-monogamy" has been one of its meanings in the past. But I think it's more standard now to think of polyamory as a *subset* of responsible non-monogamy.
To distill all the varying opinions about the definition of polyamory, and favor what I perceive as the most common definition, I would say that "polyamory" means: "romantic relations involving more than two persons, with the knowledge and consent of all involved."
One of the big deals about that definition is the word "romantic:" It implies a relationship that is both sexual and emotional. Swinging, for example, tends to not have that emotional element, as it complicates things to "fall in love with a play partner."
From your description, it sounds like you prefer to keep your relationships pretty light. Almost like a "single but with friends-with-benefits" type of a thing. Therefore, you're teetering on the edge of what some people might call "poly," but you're well within your rights to call yourself "responsibly non-monogamous" or "single with friends with benefits."
Ultimately, these are all just words, and people don't always agree about what any one word means. Plenty of people are uncomfortable with "labeling" themselves with any word, and you're not under any obligation to wear a label. I'm sorry if people have made that a difficult issue for you; they shouldn't have done that. Still, some good comes out of it if it leads you to understanding certain words better. Understanding words (and their varying definitions) is part of what helps us become better communicators.
It's completely up to you whether you decide to wear this "poly" label. Like I said, you're just in that in-between area where "poly" could fit but it doesn't have to. It's just a word, so if it helps put other people's minds at ease, and thus helps put your mind at ease, by all means make it a label and wear it proudly. But don't feel that you have to; it is purely a personal choice.
I hope some of this is helpful to you. Please let me know if you have further questions, concerns, or comments.
Glad to have you aboard,
Love means never having to say, "Put down that meat cleaver!"