He is your first, you said, so you are looking at your situation colored by some sort of romantic idea about that. But there are other men out there! There are others who would love to respect and cherish you, and work toward building a healthy relationship with you.
Don't settle for this jerkwad who only cares about himself and no one else! He doesn't really know what poly is, and I say this because he doesn't give a shit about taking care of the people in his life or, at the least, working towards a consensual arrangement/agreement with his lovers.
My advice is to end it and move on.
I am sorry, that probably means going to that event might be too raw this year, but you need to do what you can to preserve and boost up your sense of self-esteem and self-worth and invite beauty and joy into your life instead of all this pain -- and being treated so callously.