Sit him down and talk about it. If Sunday nights are supposed to be just for the two of you to bond, then you have to address that issue. All the NRE (New Relationship Energy) in the world doesn't excuse him for blowing that off for a new person if you've made it clear that's a requirement for your relationship.
When you do discuss it, to avoid sounding like a crybaby involves not sounding like a crybaby.
In other words, state the situation clearly, state the problem you have with it, then ask if he has an issue with that, and negotiate how to deal with such things in the future.
"As I see it, you blew me off on Sunday. Sunday is supposed to be dedicated to us being together. Spending that time with X--particularly without warning--appears to me to be disrespectful. I was hurt by your actions. I would like to avoid hurt feelings in the future, so I need to know what you were thinking, and we need to figure out what we are going to do."