It also sounds to me like confusion over terminology and definitions. In the end, a term is only a tool to describe something, NOT a box to fit into. There are so many different variables and situations in relationships, it always seemed silly to me that people would try to fit into one of a only a few categories and depend on those terms to tell them what the relationship should be, and what the boundaries should be.
Much better to define for yourselves what works, what is happening, and what you are comfortable with. Don't let assumptions about what a term means lead to misunderstanding or allow agreements you are unaware of. (lets face it, not everyone has the same definitions for things). If you don't think what you did was poly and he does, then it sounds like you are due for a conversation about what you consider poly.
From an outside view (just my own) what you are doing right now sounds to me like "swinging" with a poly option. I'm not entirely clear how people in the poly community categorize swinging, but the swinger lifestyle participants only rarely consider their activities polyamorous. Again, a lot of it just what you choose to call it. Two couples with the exact same behaviors may choose to call it different things. I think there is a difference between poly and swinging, at least, to me there is.
I'm concerned about your use of the word slutty - it usually has negative connotations. Maybe it doesn't for you, I don't know. Whether or not you and your partner agree about what you choose to call your activities, you should NOT be doing things that make you feel bad about yourself. If you feel like its wrong or it makes you uncomfortable, it doesn't matter if its poly or not! There is always more than one way to do something, and if this particular aspect doesn't work for you, you can cut it out! You don't have to accept it as a necessary evil or just the way it works.
It's up to you to define what poly is for you, and not to let the term poly define what your relationship should be.