I can identify with this. When my wife and I were starting out with nonmonogamy I was looking for something closer to soft swinging with good friends, as I'd experienced in college when single, which my wife thought she could be ok with but didn't find appealing for herself. We used the term poly because I didn't know any more accurate terminology (soft swinging), and it seemed that polyamory was an umbrella term that people defined the limits of themselves. Then someone from our group of friends started talking to her very often and over time it became physical. I provided my full support and was glad to see her beginning to realize that it was possible to be physical with other people and still be close to me.
I hadn't thought about how I'd feel if she became more than physical with someone else since she'd made it clear we were not to develop romantic relationships with other people. There ended up being some confusion as in the midst of being pursued she forgot that she'd made it clear to me that we were only to be physical with other people, not romantic, (her rule initially) and mistook my support for her being so physical as support for developing a relationship. Developing the relationship was what made it all seem worthwhile to her, the benefit to nonmonogamy, so since then (a year ago this Monday) I've been seeing how well I can adapt, dating others when before I thought I'd just be more affectionate with friends, and becoming more active in organizing poly events.