I wish I had an answer, because I'm in a similar situation! I would LOVE to live with both my husband and my partner, but I really don't see that happening anytime in the next, oh, decade (having kids is part of it for us, along with lack of space!). TGIB (my partner) and I have talked extensively about to what degree we want to entangle our lives, and while he, like me, would like more than what we're currently stuck with (LDR) he knows he needs his own space and once he’s out of his current situation does not want to live with ANYONE for a while. At some point he may end up with a roommate or living with an OSO and that will be hard. I know I'll be jealous because that person gets to be there with him every day and I don't. I’m hoping, though, that we can work out a living arrangement in the next few years where he’s in an apartment over our garage or in some sort of in-law quarters. However, that’s only a possibility because of our particular situation and set-up. I doubt it would be an option if he had other partners already.
It may be that I, too, am bad at being a secondary- when I'm close to someone I want them around! Maybe not 100% of the time but I want them around more often than not! Unlike what seems to be a lot of poly people, I don’t like alone time (I’ve never even lived alone). The things I do to recharge still involve people, just maybe different people from the ones I’m usually around. I’m not poly because I want to be independent- if anything I’m poly because I get close to people and want to make them an even bigger part of my life.
I know with kids it can be VERY difficult to do overnights but once a month sounds like a good starting place, and then gauge if more is possible/needed or if fewer are necessary from there. However, I don’t know that your relationship necessarily needs to “grow” from the point it’s at now. I guess I don’t see how sustaining a relationship necessarily requires growth. Change, yes, but that happens automatically with time, regardless of the circumstances. Adapting to the change and keeping your connection is what will sustain your relationship, as I see it, not necessarily growing beyond what it currently is.
Again, I wish I could be more helpful but you are definitely not alone in feeling the way you do!
Pan Female, Hinge in a V between my mono (straight) husband, Monochrome and my poly (pan) partner, ThatGuyInBlack