Vicarious, I am myself still new to world of polyamory so I don't think I can have any advice for you but I just wanted to thank you for sharing your story cause its kinda similar to mine. I too get aroused by the thought of my wife being a slut and I frequently masturbate imagining her doing it with other men. I thought that I might be alone in that but I'm glad to see that there is other men out there who share this, kink. My problem in this however is that the man my wife has chosen currently to be with is someone I find very repulsive physically so thinking of them together has quite the opposite of arousal feeling for me. More on this in another thread that is awaiting moderator approval.
I also share this "Can't really have arousal when it comes to other women" thing. I'd rather she take other partners and would really love for her to have partners that share this kink of mine and would want to share her in bed at the same time.