He can go by himself, while you do something that appeals to you or visit people you enjoy being with.
Personally, I choose not to get onto angst-y situations and hang out with people whose company I do not enjoy, or against whom I will be defending who I am or how I live, just because of some arbitrary day on the calendar that society decides should be celebrated in a particular way. And I don't feel that family should get a spacial pass to take shit out on me just because they are related somehow. So, I'd be saying, "You want to go, fine - go, have a good time, but I'll be at such-and-such having a good time on my own. See ya when you get back." Boom, done.