With the background you've provided, I think it all sounds very sensible and like it could totally work out! And, importantly, it also sounds like if it didn't, it wouldn't be an irredeemably terrible mess. For what it's worth, I say go for it.
One final question -- he's said he'd like to have your kid, in so many words. How do you feel about that prospect? How does your husband feel?
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.