I am not sure how many of you have read the book Sex at Dawn, but it covers the prehistoric underpinnings of this phenomenon, 2 or several men all involved sexually with one woman.
It posits, through biology and anthropology, that a woman having sex with several men in one session (over several hours or days) is completely natural. I am assuming the OP does not just watch his wife have sex, and then not shag her himself. He watches, he becomes aroused, he also has sex with her shortly after the other lover is done. Women can be quite multi orgasmic, and in general, men are not as orgasmic. So, she is biologically able to have sex with more than one partner in one session.
The book also suggests it is one reason women are often quite vocal during sex. Her noises (in pre-history and pre-patriarchy) would act as a mating call to other partners to "come and get me while I'm hot."
If you are not culturally programmed to be jealous of another guy boning your wife/gf, why wouldn't it be arousing to watch her physically involved with another partner? It could even be a socially bonding thing between the 2 (or more) males involved.
I think Sex at Dawn is a very interesting book, and I hope other polyamorous people find it as enlightening as I did.
Love withers under constraint; its very essence is liberty. It is compatible neither with envy, jealousy or fear. It is there most pure, perfect and unlimited when its votaries live in confidence, equality and unreserve. -- Shelley
Mags (poly, F, 60) loving miss pixi (poly, F, 38) since January 2009, living together since 2013
also loving Punk (monogamish, former swinger, 42, M) since Oct 2015
"Master," (mono, 34), miss pixi's Dom for 2 years