I think it is easy for people to people to show off their prejudice for something different from they are use to. I have never seen anything to show that polyamory hurts the children. But truthfully, there is very little research at all. But that does not stop a social worker from assuming that it must.
My guess is that kids are not harmed unless they constantly are losing adult figured in their life. I think they may get a bigger benefit if more adults watch over them and care for them. Plus I think it is neutral for the parents that hide it from the kids.
I agree with avoiding the therapist. It sounds like they are being too judgemental without evidence. You don't need that in your life.