For me, sex doesn`t come into the equation of wether someone fits into the poly category, or one of the others.
It`s strictly about my feelings. Do I feel something for them that is romantically inclined ? There have been people I cared for, felt that spark, that chemistry, that for one reason or another, I was unable to persue the romantic interest.
Knowing I felt what I did, means I didnt have them in a 'friend' category.
I like being a voyeur. I also like directing. So for me I can be in a sexually charged environment, and as long as someone isn`t putting their hands on me, I am fine watching others. I am careful not to give mixed signals to people. I try and respect other peoples boundaries as well when in those environments.
I guess for me, it boils down to touch, and romantic interest. I have to be romantically interested in someone in order to be touched by them, or want to touch them. They can be as sexual as they want with each other, as I only care to control me.