Ouch, I feel for you! It sounds like you two have a lot on your plates, and maybe he is even a little depressed?
If he seems to be in a place where "yes" is the only answer he wants from you, try this one: "Yes, I agree you need some romance and dating and sex in your life, but with our toddler and this pregnancy and everything else, I'm worried you don't have time for something like that. Can you prove to me that you do? Start by dating me. If you take me out at least one night a week for the next few weeks, and spice things up in the bedroom, and then you still have time and energy for another woman, we'll talk about it then."
And who knows, maybe once he is meeting all your needs, you won't feel so strongly against letting him "off leash". (Or maybe not.)