I think there might be the idea for some of them that if your wife doesn't now and you are wired in a monogamous way, they wouldn't be "sharing". The wife would get the bad parts, they'd get the good ones. Also, potentially, they might be able to "get" you eventually, since cheating would mean there is a problem with your relationship.
With polyamory, even if they have you, your wife does, too. If they want to truly "have" you, paradoxically they might feel it is more the case if you are cheating than if you are not.
I too have the problem of "I'm polyamorous" "Oh, when can we meet up" or "Oh? Er, I'm sorry, but I'm not interested".
I don't know why they always assume I want to have sex with them. They wouldn't assume it if I was single, I'm sure...