you should research your state's laws on child custody issues. i know in my state custody technically will go to the parent who is not or has not committed adultery. i'm in north idaho and it's very conservative here and very old fashioned in state laws even though the amazingly liberal WA state is right next door.
no matter what any of us say here, it's up to verify exactly what you are up against. i do think if he tries to use polyamory against you it's going to back fire since he's "leaving you" for the "other woman". he'd actually need to prove you are unfit - dirty house, dishes never cleaned, piles and piles of grossness, drug use, abuse of the kids...that's what will help him get 100% custody.