Where do you live? I am interested because I am attorney. You do need a good lawyer. Most will be scared more than likely because of not understanding poly. You should research laws concerning same sex relationships and the rulings of the appeals courts in your state. Or any other non traditional relationships. It will give u a good indication of how the court will treat this.
You do have an advantage because you have temporary custody and there is supervised visitation. You are ahead at this time. Also because your husband participated in non traditional relationship you are at an advantage. He gets less traction because he participated also at some point.
Finally I would have kids testify to prove they are not negatively effected.