This resonated with me, BG. My partner obviously limits the time he spends with me (I'm a secondary, he's the hinge in our Vee). I am sure the time we spend together is a topic of conversation between him and his wife. (In fact, I know it is, since when we plan things out in advance, he usually says he needs to check with her calendar.)
He presents it to me as his choice--and whether or not his choice comes from an agreement between the two of them or not, it IS his choice. In six years, I have never heard anything like "my wife says we can only...." or "I can't come see you because my wife...."
Frankly, I'd be hurt and pissed off if he did. He has free will. He may choose to agree to whatever limits they set between themselves--but I have no need or want to know about those agreements, which are between them.
I know my partner works hard to advocate for BOTH his relationships, and also to have enough time for himself. And that's all I need to know.