I am really disturbed by the fact that he doesn't seem to be that much involved in the everyday things around your relationship. I mean, a child with such a serious disorder should fill your head completely! That there is room to think about something like adding more to this situation, seems really odd to me. Is he escaping from the situation by acting the way he does? All his concern should be with the unborn and your health.
Stay strong and stand your ground in regard to him working on your relationship first. It doesn't seem solid enough to withstand the rocky path a start into polyamory tends to resemble.
Facts: 30, female, bi, v-type relationship with Sward (husband, straight, mono) and Lin (boyfriend, straight, mono), poly-fi and co-primary.