Wow, my own story parallels yours in so many ways. Closer friends with women than men, those relationships often growing to a feeling of romantic love, not wanting casual sex (although I wasn't often self-aware enough to tell just WHAT I wanted). Not caring if my wife was with another man, even wanting her to be.
Your situation is much better than mine presently, so I will offer some of mine as a cautionary tale. I felt the way you do for many years, but I was committed to monogamy with my wife, and tried to set up defenses to keep me from being tempted to cheat.
I stopped spending time with women I thought could turn into close friends (because the temptation to become lovers might be too hard to handle).
I stopped sharing my poly desires with my wife. It was clear that it hurt her, and I resolved to just grit my teeth and bear it.
These things were my own ideas, done of my own volition, and I didn't share what I was doing, or the reasons for doing them, with my wife. Looking back, I can't believe how stupid I was.
Eventually, I became friends with women I knew in an online game, and suddenly I found myself cheating on my wife. Consumed by love for my new partner, feeling a greatly increased sense of love for my wife, and hating myself, really HATING myself for cheating. No physical contact, but still an emotional and sexual betrayal.
It sounds to me like you are closer than you realize to cheating.
I certainly wish I had managed to talk to my wife about this; I still really haven't, and she's now my ex-wife. Its only since our divorce that I really understood myself to be poly, and not just an asshole who couldn't keep his dick in his pants.
If I had talked to her, I strongly suspect that our marriage would have ended anyway: she was mono, I was poly, and we both couldn't be happy otherwise. But I could be completely wrong. I can't go back and find out now.
Even in the worst case scenario, if I had talked to her and our marriage had blown up, it would have been better. Better for my own self-esteem, better to have avoided the wasted years we both had in a marriage that had lost its romance, and better for our future relationship as friends. Better to not have hurt my wife so badly.
But your situation is not mine, so I'm not offering advice exactly, just my personal regrets.
Good luck man.