I'm new to poly, but my understanding is that honesty/consent/respect for each other's boundaries are central to making a poly relationship work.
By your account: he was not honest with you about seeing other people (a lie of omission is still a lie); while you did consent to a poly relationship, you did not consent to him hiding his other relationships from you and abandoning you while pregnant to be with one of his other partners when you needed him; and he did not treat what I am assuming are your boundaries (honesty, being there when you need him) with respect.
Abusive or not- this is toxic.
Furthermore, he sounds like he lacks the emotional maturity to have an adult relationship, regardless of whether it is poly or mono: he doesn't validate/show empathy for your feelings; he shuts down communication if it is about something he doesn't want to hear; he FREAKING LEAVES YOU TO BE WITH ANOTHER PARTNER WHILE YOU ARE PREGNANT AND NEED HIS SUPPORT (sorry that one is causing me to get mad on your behalf, if you impregnate somebody, you should be available to support them through the process); and instead of accepting responsibility for his actions, he blames/criticizes you for your normal reactions to his abnormal amount of bullshit.
Also, let me guess, the couples therapy he wants to go to, it's supposed to help you adjust to/cope with the way he is behaving instead of teaching the both of you how to communicate assertively, reflectively listen to one another, and work towards win-win situations, isn't it? If I'm right, that's just another way of him to avoid accepting responsibility for his actions and avoid sacrificing anything to make you feel more secure and valued in this relationship. <ahem> emotionally immature <ahem>
The icing on the cake is that he had the nerve to say you are playing mind games. Honey, that is called "projection", instead of acknowledging that he is the one who is fucking with your head, he tries to pin that on you. Or he could just be delusional and actually believe that he is in the right...
Either way, is it worth it? As the mother of my first long-term boyfriend once told me, "You never want to raise your husband." That goes for primary partners too. You can't force him to wear his big boy pants and act like an emotionally mature adult. That's for him to come to terms with, if he ever does.