So how old are the kids (his)?
What does he want out of the relationship?
What do you need to feel secure about the relationship? One date a week? Every other week? You may want to consider what you would like to see happen and propose it to him.
It sounds, just from listening, that the idea of living together was a bit of a pie in the sky musing on his part, a daydream. But I'm just guessing.
I remember, in our quad, hubby and I came to the point in which the household drama (roommates, extra kids in the house, etc.) took over the lives of our OSOs, and we felt "left behind." It took a very concrete conversation about what we wanted to make progress.