I agree that it's a parenting issue. Just as many monogamous single parents allow a stream of people into their children's lives. It's simply more understandable for a monogamous single parent to do that because of course, we are all looking to replace the child's biological parent in the child's life as well as our own. So as bad as it's viewed, there's a lot of pity for poor single mum who got dumped by her baby daddy and now she needs to find a new daddy for her bastard child.
The desire for non monogamy is already seen as pathological by many, so it's not just "single mum", it's "crazy nymphomaniac single mum whose baby daddy had a lucky escape seeks new victim(s) to help her fuck up her kid".