I can't imagine how hard what you are going through must be to cope with.
I think that decisions around children are so hard in general. For women, it's not as easy as saying wait until things are sorted out and you're in a good place. Sometimes time runs out if you do that.
The stress of what you are going through just now must be hard. Like Phy, I wonder if what your husband is expressing is a way of avoiding dealing with what is going on? It seems like such a strange time to be starting discussions about opening up your marriage.
Do you have other sources of support that you can lean on as you deal with your pregnancy and caring for your toddler? I hope you are able to look after yourself both physically and psychologically.