It does sound like the OP is more interested in watching or hearing about his wife having sex with other men than in one or both of them pursuing intimate relationships (i.e. falling in love) with other people. While this "voyeurism" is not "anti-poly" is it not "poly" in and of itself.
Ok, folks. This is the part where you berate me for "judging" people.