The book "Sex at Dawn" mentions that "watching a woman with other men" thing, and how it's based on instinct. If it's about the sexual aspect only, I would say it is swinging territory. If you think that want for her to be happy and fulfilled, be it with you or someone else, can extent to emotions (as in, watching her be in love would make you very happy), then that would make you a "poly-friendly" partner, although you seem mono yourself.
This being said, it seems from your post that it's only on a sexual level.
I can't say I find your case weird. Not only do I know many many guys in your case, but "Sex at Dawn" explains how it came to be (you mention it in your introductory post, the fact that the species is adapted to females having multiple partners, and so on).
I think one thing here is important though: you would enjoy it, but maybe she wouldn't. It is your fantasy here, not hers. While it's good to make her understand you're fine with it, she needs to know she doesn't have to ever do it, and you need to accept that maybe she doesn't want to. You said that you have no interest in doing the same thing, after all.
Also, you need to keep in mind that there would be a third person: the other man. Someone who has feelings, expectations, etc, and might not want to feel used as a sex toy.
And finally, there is the possibility that this arouses you extremely but that you would actually hate it f it happened. While you seem to be very aroused by the idea, it's still possible that seeing it would awaken weird, conflicting emotions. So I would say, make sure you're comfortable with it 100% and it's not one of these fantasies that are only arousing as long as they stay fantasies.
Otherwise, I don't have much advice for you... And polyamory focuses on the emotional aspect of things so I would say you could get more advice in another forum that is more sex-focused (not that we don't talk about sex here, mind you!)