I gather you work evenings. Something to consider as well is your energy level. When my energy level drops, I am less able to cope with EVERYTHING. The thing of it is, if I fail to realize the creeping tiredness - or slight but ever-present chronic tiredness that some shifts bring - then I will damn well find a reason why I feel upset, when really all I am is tired or under-powered.
Since you feel okay when it is the three of you, I am guessing you are getting energy from the group dynamic. More energy - problem solved! But if you work a shift that is more stressful to you, and work - regardless of whether we like or not - requires your personal resources. Of course you'd rather be home with the people who love you.
Hell, I have even volunteered to work shifts for people who had a family emergency, and still found myself yearning for home, even though I had volunteered!
So I am guessing the core issue is that you wish you were home, and has little to do with the relationship dynamic between the three of you - which sounds pretty healthy.