Anabelle, thanks for the advice!
Yeah, I guess you were right - in attempt for my wife and I to find a compromise we totally overlooked the other person's interests and feelings.
Maybe even more importantly, I realized that maybe I was splitting hairs over the affection and love, either way its having feelings for someone else. I guess that means I can have feelings for more than one person and that makes me poly.
Anyways, it felt pretty good owning that and explaining that to my wife. It was a bit intense but I think we are cool with it and it actually made me even more positive about my marriage too.
We haven't renegotiated the rules but I see how the one-night threesome thing is kind of creepy. I guess I will focus on one-night stands for now and try to renegotiate in the future if I feel the need.