I get what you are saying, "why the change, when there was nothing obviously wrong?" I can't speak to your self worth as others have. It may or may not be an issue - you are in a better position to judge that. Regardless, an unexplained change in behavior would leave me very puzzled and hurt also - and I assure you, I have no self-esteem issues.
That said, the others have given you good advice. Move on, and do not let this man back into your life. I could hypothesize a number of reasons for his behavior, but none of them are a reflection on you.
By the way, that whole passive-aggressive remark to you was projection - he was projecting his own behavior on to you.
Hang in there!