Maybe you'd have less of these negative feelings if you could set firmer boundaries about how you interact with others. It sounds to me as if you are more polysexual than polyamorous although you do enjoy connecting with people in other aspects.
You can absolutely live a monogamish lifestyle without it being about possessing another person. It's just about having a partner that is happy to have those boundaries in their relationship and from what you've said about your partner having less attachment to the other people he has sex with, I think he will probably be fine with that and also be able to keep to those boundaries without changing how he has been doing things thus far.
I think if you could be more certain that you're not going to end up in a full on polyamorous situation, you'd be able to enjoy things like sharing explicit details more often as well as anything else you like about having a non monogamous relationship.
Polyamory isn't the pinnacle of ethical non monogamy, it's just one variant.