I may be wrong but I truly feel that there needs to be a sexual buzz for her when you see this women for it to work. My husband and I talked/fantasied about him having sex with another woman (we've been married and monogamous for 18yrs). We have now done this with a friend of mine, but I am there, I enjoy being in another room and then am required to join them. In other words we are 3 equal participants. The correct term for me is cuckquean (although I hate the term) it works as I am sexually submissive and enjoy/need the 'humiliation' my husband is dominate and gets pleasure out of my demise (in a good way). I have no desire to seek sexual encounters on my own, but it really works for us.
Just a thought, but maybe your wife would like to join in but can't admit this to you or herself? With me as the primary encourager they had sex while I was not in the house but text me with what they were going to do. I have never been so turned on in my life.
I also know though that if it was upsetting me my husband would stop as he says if it doesn't do it for me, it won't do it for him, we are equals in a loving relationship and our 'friend' is as close to me as him.