There was a thread that covered some of this here: http://www.polyamory.com/forum/showthread.php?t=339
I think it comes down to the state and the judge. However, I think that if you are a good parent and the kid is doing well, then you should be ok. If something did happen, there are a few groups that would defend you if it did happen.
There was a case about a decade ago where a woman appeared on MTV with her husband and live in boyfriend. Her child was taken away. She received a lot of money to fight the case. I believe that in the end, she gave up because she admitted that she did not have the finances to take care of her child.