There's no reason that "it's just me" can't also be "something I need to talk to her about." The key is that you're sharing your feelings, not asking her to stop seeing him.
A metamour whom you don't like is not different than a partner's friend whom you don't like. No one's asking you to go out with them, so don't worry about it.
All grown-ups should be capable of acting civil when in group situations with people they don't care for. This applies to friends of friends, friends of lovers, lovers of friends, and lovers of lovers. It also applies to your boss and that weird guy in cubicle 3B. And boy, does it ever apply to mothers-in-law.
I would suggest letting your partner know, tactfully, that you'd prefer not to spend time with the guy. Make it clear that you have zero objections to her spending time with him, you just don't want to participate. There will be situations where it's pretty much unavoidable, but New Years and her birthday only happen once a year.
Gralson: my husband
Auto: my girlfriend