It sounds to me like maybe he is setting himself up to be the VICTIM. And setting you up to be the "bad guy." When in truth, he feels like a failure -- but he'd rather be a victim than a failure.
It's black and white thinking, someone has to be right and someone has to be wrong -- instead of, something is missing in our relationship, what is it and what can we do about it?
It almost sounds like he is hoping you WILL fall in love with someone else and leave him, kind of like a passive way of escaping his problems and his responsibilities in life. He doesn't have to do it this way! He is scared. I think he's afraid he can't make you happy and he can't take care of his family the way he would like. Financial fears are real, I get that, but it sounds like he is frightened of intimacy, as well.
If he won't go to therapy with you, I always hear professionals suggest that you go alone. At least you can get some help sorting through which issues are yours and which are his. Sounds like he's blurring them. You deserve some clarity!
Formerly married to Sundance
Boyfriend -- Butch Cassidy