It seems to me he doesn't want primaries at all. Only secondaries. Whatever the reasons, it might just be how he is, and he was clear to you about that.
You said you were fine with it so he'd have you back, and I would say, at that time he had made his choice: he wanted to be in a secondary relationship with you, or not at all.
I believe you have a choice too: accept a secondary relationship or leave. I don't think he'll ever change his mind, and hoping for the situation to change will hurt both of you.
I don't think he would leave you for the other women, considering he cares a lot about you, but his condition is for your relationship to be less involved and committed. If that doesn't work for you, I would say you need to end it.
If your issue is your insecurities, working on them might help, but I personally don't think you will change either. It's not like either of you is the problem, both your attitudes are fine, you just need to be honest about what you want. It seems to me you need a different kind of relationship than the one he is willing to offer you, and he needs a different kind of relationship than the one you are willing to offer him.
It's not about polyamory in itself, as the level of commitment in a relationship isn't tied to that. He could be with only you and still want to be more casual than you'd care for. And there are many poly relationship that involve the level of commitment you are looking for, either with several primaries (but it doesn't seem like that would work for you) or with just one (which would be your ideal scenario from what I understand).
I suggest you talk to him about that and make a decision (stay with him in the kind of relationship he's offering, or leave) and then stand by it without regret because you'll know it was the best decision to make.