When I mention that we are non-monogamous to people who are in a monogamous committed relationship, they tend to get this expression of doubt on their faces, or even say words to the effect of, "*I* could NEVER do THAT. MY marriage would be OVER if THAT ever happened." Or, the converse reaction, which is, "OMG you are SO LUCKY! You can go out and get laid any time you want!"
So, I think one reason people have a problem with their family members' non-monogamy is that they see it is a threat to that which they hold near and dear. They don't care so much about strangers, but if it's your RELATIVE, this might be HEREDITARY, and they might GET it some day too.... much like some schools of thought say that homosexuality has a hereditary component, and when a child is gay or bi, phobic parents are afraid the kid "got it" from them (my mother said to me "I was attracted to the tennis instructor once a long time ago; maybe that's where YOU got it from")
and they feel that it will jeopardize their world view, comfort zone, and everything they've worked toward building their lives as they now know.