Blondie2, polyamory has about as many definitions as practicioners. Personally, I tend to consider polyamory as multiple relationships with emotional substance rather than multiple sexual partners. I'd suggest that multiple sexual partners without the emotional connection is swinging, not polyamory. Whatever your definition or the definition or your partners, the important part is that everyone involved understands what is going on. I know many folks who use the poly label as an excuse the sleep around while keeping a spouse at home but never actually having any real connection. Some poly folk define cheating not as a sexual act but as the act of pretending something true is not true.
The fact you're frustrated at the flirting comment can be a good thing. If you sense something us out if place, then there's an area to pursue. Why dies he say it's great, for example, is a simple question for him. A simple question for you is what are some specific things that have confused you?
Too be fair, being poly is not a good idea for the faint of heart. Success requires a level of introspection and generosity that many struggle with. The fact you've questioned how to handle things is a huge, strong step though.
There are two questions that have been excellent guides for my relationships (poly or mono): 1) Is this a level if crazy I'm willing to deal with? Much of what we face in life is based on our choices. For example, I've had friends who were downright toxic to me. As long as I chose to be near those folk, my life was full of unnecessary drama and angst. Sometimes the added complication is worth it, sometimes it's not, but go into it or away from it knowing that the choice is yours.
2) If there is no good answer, should I ask the question? Sometimes we seek things that do nothing but upset us when we would have been fine without ever asking. The classic example is someone asking, "Do you live me?". There are two answers: yes and no. If they say yes, why did you have to ask? Feeling insecure? Jealous? If they say no, what now? Were you not happy without having to articulate the question? Did having an answer actually help?
Best if luck to you and yours.