I operate in a lot of "Alternative Parenting" circles. In these situations I've learned to stand up for my point of view without engaging people in discussions or conversations about issues. DO NOT try to defend yourself, because this gives them an opening to argue with you, and makes them feel like they may be able to prove to you that you are wrong. So just don't go there. As we say in my alternative parenting circles "Bean dip 'em". This is to say CHANGE THE SUBJECT! "Hey mom/dad, we're poly, I have a boyfriend in addition to my husband. Would you like some bean dip?" When they bring up the "your kids knowing your boyfriend is a problem" thing just say "I'm sorry you feel that way, I disagree and as their parent that is my decision to make. Would you like some bean dip?"
You can use this for any and all issues where people would like to try to convince you that your point of view is wrong. With this in mind, my partner is a male to female transexual. This is not something we could just NOT tell our parents. When my husband disappeared from family photos, and images, and life in general to be replaced by an eerily similar wife ... that was not going to go unnoticed. So, I wrote an email to my family (I have a HUGE family, I have 7 siblings most married with kids of their own, this is how we all communicate) and in it I said. 'S is transitioning to a woman. This is a brief summary as to why. This is happening. If you're curious and you want to know more please feel free to ask politely. If you have nothing nice to say please do not respond. If you say anything negative you will be ignored.' And that was that. I have not heard one negative thing from ANY of my family, despite the fact that they are Mormon and this is "wrong" to them. In fact I've barely heard from them at all. That is their choice, but I will not take their crap and let it make my life negative.
Stand up for yourself, state what you will not accept from them, and "Bean Dip 'em"!