It sounds like this is all very focused on the sex. Which is fine, but that's not all there is to polyamory.
If you want to have a successful polyamorous arrangement with the three of you, then take the time to get to know each other better through other, non-sexual activities. Also keep talking about your feelings with your husband. If you feel concerned about your daughter, then it would be good to limit the times she is around her - many polyfolk do not allow their children to know other partners until they are sure they are trustworthy and going to be there for a long time. But children can be, and have been, successfully raised in a poly environment and be quite healthy and open-minded because of it.