I agree that it's not unreasonable to just explain how it will make you uncomfortable and if possible to work with you on that. You could even make an offer to give a quick call when you are on your way home (15 min notice).
And in case it might be helpful in the future, what we've found works for us for the most part is just to respect what we walk into and assume if we are wanted - we'll be invited. It's almost as uncomfortable for those involved as it is for the one who isn't ! Not wanting to hurt feelings or exclude someone - but yet savoring the private moment.
So having a mutual understanding that private time can be important and that there's no obligation to invite unless the mood is right just seems to put everyone on a more even keel.
Play time all revolves around moods and it's important in the long run to make space for all the mood variations. Being able to be 'private', to decline etc without it becoming personal is key.