He can go by himself, while you do something that appeals to you or visit people you enjoy being with.
Personally, I choose not to get onto angst-y situations and hang out with people whose company I do not enjoy, or against whom I will be defending who I am or how I live, just because of some arbitrary day on the calendar that society decides should be celebrated in a particular way. And I don't feel that family should get a spacial pass to take shit out on me just because they are related somehow. So, I'd be saying, "You want to go, fine - go, have a good time, but I'll be at such-and-such having a good time on my own. See ya when you get back." Boom, done.
The world opens up... when you do.
"Oh, oh, can't you see? Love is the drug for me." ~Bryan Ferry
"Love and the self are one . . ." ~Leo Buscaglia "