Polyamory has only recently come to the attention of the scientific community, and there aren't a ton of studies at this point about how children are affected. The few that have been done show similar findings to those done on children raised by homosexual couples (of which there is a substantial amount of research). That being, in itself, polyamory has no affect on children. In studies done on families with homosexual parents, the only negative difference found is that children of gay people are exposed to more intolerance from individuals who do not approve of their parent's relationship. It's likely that further research will show similarities with the children of poly parents.
In short, it's a "people are ass holes" problem, not a "polyamory isn't healthy for children" problem. Not being polyamorous won't change the fact that some people are just dicks.
As far as Granny is concerned, I would bet a significant amount of money that if the poly thing wasn't an issue, she'd find something else to be upset about and still cause problems.