I thought it was always your agreement that your wife spend christmas eve and christmas day with you and boxing day with the other guy? So I don't really see how he has been the bigger man? After-all as you say when she's with you she's shared with you and all the kids; when she's with him there's just the two of them. You are one of my big mono heros the way you cope with it all.
If I had the feeling that Z was "elsewhere" when he was supposed to be with me sharing a very special time I would be furious. Although from her point of view I can understand that she was probably missing him especially if he was alone. Maybe it's time to think about sharing your space and would be better next year if he came to you?
I hope you will get some one on one time with your wife these holidays because that time is just so vital and precious for a relationship.