Further thoughts... possibly the best way to phrase this would be to focus on what you need for your family rather than making it about the girlfriend being there too much.
In other words, you need (e.g.) a couple nights a week and one day on weekends that is designated family-only time. Then it's in no way about the girlfriend or your wife's polyamory, and in every way about your family's needs. Rather than being exclusive of the girlfriend, it's inclusive of your family. Point of view is everything.
Gralson: my husband (works out of town).
Auto: my girlfriend (lives with her husband Zoffee).
The most dangerous phrase in the English language is "we've always done it this way."