Most of the time when folks are hollering "What about the children?" the issue at hand has nothing to do with the children. They're trying to offer a rationale that has more legitimacy than a simple "We don't like that."
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.