Yeah, the, "You're traumatizing the children by removing an adult from their lives," doesn't hold water. Not unless it can be established that the child is a lot more attached to the poly adult than that same child would be attached to one of his/her teachers.
As a piano teacher, I spent 30 minutes a week one-on-one with many children. After months or perhaps years of lessons with me, the child would move on (to another teacher or to another field of study). It was more than just a teaching gig; my students and I became friends in the process. I miss them today, and flatter myself that they miss me too. We all have people come into our lives and then go out again. Missing them is part of the experience of living.
Just think of how the series M*A*S*H ended. Hawkeye and B.J. -- inseparable friends -- said goodbye to each other -- for good. And think of how Lord of the Rings ended. Frodo departed on a ship to the Undying Lands, in the process bidding Sam farewell -- for the rest of Sam's life.
It happens to all of us. We lose people. We miss people. And yet we treasure the memories of good times we shared with them, and life goes on.
With polyamory, I think you have to examine each individual case and judge it on its own merits. You can't just say, "Well you're going to damage the kids every time they lose an adult."
Love means never having to say, "Put down that meat cleaver!"