Definitely be super aware of your safety right now. He may/may not have borderline personality disorder, but since you have broken it off with him I'd behave as if he does and take precautions for your safety as if that's the case. Better to be safe than sorry.
As for the rest, I think you need to stop trying to make sense of what he says. He obviously has a huge amount of mental and emotional issues, which means his opinions about what happened and what he says have a tenuous basis in reality at best. Maybe if you look at him like someone who's so caught up in his own world that he can't see what's actually there, you won't let yourself get so upset about it. And you definitely don't need to spend time defending yourself.