It does make sense. It makes perfect sense. You are an accessory that comes with the kids.
How old are you? It sounds like one of those "mid life crisis must feel young again" cases. I'll bet that's what seems so appealing about the girlfriend. Is she much younger? if he thinks he'll be too old to have fun when YOUR kids are grown, what does he think will happen if he starts a new family with the girlfriend?
I still can't believe he invited you to their wedding which will be in two years...