Oh, I didn't think they were lovey-dovey conversations at all. But since he is feeling shitty that you shut down, and you are feeling shitty that you're shutting down and letting your emotions be so visible on your face, there's no reason to not take conversations elsewhere when possible while you are still feeling really vulnerable and trying to figure out what you want to do.
Then again, I'm of the school of thought that phone conversations others are having should be had out of my earshot anyway unless it's going to be short, because i find it distracting, even rude if it's a regular occurrence. If I am in the car, with a partner OR friend, I either want to be chatting with them or listening to music. If they want to spend their time on the phone, and it's not important, they can wait until they have a bit more privacy. But sure, if this is the normal routine in your house, it would be weird to change it now, but it's not like he doesn't KNOW you are having trouble dealing with the new relationship ideas he's brought up.
Happiness will never come to those who fail to appreciate what they already have.