That's really what your 20s are for -discover who you are and make your peace with that. (It's also why your 20s can really suck ass.)
This is so completely true.
Also, I know pretty much nothing about poly, but I had a boyfriend who had self issues that led to him neglecting me like that, even getting angry at me more and more frequently. I loved him so much so I stayed with it regardless, actually fought like crazy to hold on to it. Needless to say it ended pretty badly. We've both grown up a lot since then and are friends now, but if you want my advice, get out before things get worse, especially if he won't talk about it with you. That's always a sign that things are going bad. Maybe when he can sort himself out you can take another chance, but for now you're just going to get hurt, and you deserve better. Hope you figure things out.