Like others have said, calendar dates are irrelevant. Plan an alternate day for you guys to celebrate as a family (big dinner and all) as if it was Christmas Day. I always liked the day before and the kids sure aren't going to complain that they get to open their gifts early. Then just do one special gift for the kids on Christmas Day and make it a fun lazy day with just you and your babies. Make their favorite breakfast, spend all day in your pjs, do finger foods for dinner, etc. Don't think of it as a day to be alone, think of it as a day to spoil your kids and all your attention is on just them.
I'd be more worried about the fact that he's willing to ditch you on Christmas to spend the day with his parents and siblings.