I would say talk to them both (together if possible) about what time they want/need and how to schedule things in the best way possible. You say that you sometimes feel bad about how things work currently, but do you know for sure that your primary partner is bothered by it? Talk to him! Then everyone can be on the same page about what you all want & what is expected.
We have a schedule - slightly different situation as I live with my primary partner (my husband), but we have two days a week where we are scheduled to see our other partners (my boyfriend & his girlfriend). Some of these days are as separate couples, some are together socially as a four. Some of them are overnights, some not. It's all planned on the schedule.
The important caveat for us is that this is a minimum, a commitment to when we will always make time for them (barring exceptional circumstances, obviously). We actually often do things over and above that minimum, depending on work rotas or planned activities etc.