It's a bit of a tough situation to accommodate both sides from a position of having emotional attachment to both. That you're playing the middle person is a choice you're making to keep the peace, and that's probably the best you can do.
Speaking from father's point of view, I've had people in my life that infringed (or seemed to infringe, at least) on what I considered my absolutely sacred, scarce personal time with my children and partner - and I never reacted well to it. And never will, frankly. Anyone who is an outsider, friend or acquaintance, only gets so much time to be around on top of my time with the children. I don't share all of it, and never will.
It has never, ever had anything to do with my feelings about being poly, my relationships, or anything else. It's simply that I spend massive amounts of life energy with outsiders to provide for my family, and when I'm home with them to bond and be a father, that's important to me, and is the core of what fuels me to keep pressing on.
Don't get me wrong, there's always time that can be made for others with the children alongside - but at some point everyone else needs to go away unless they're part of the inner circle. Your husband might be the same way.
My two pence.