Sorry my friend, but it just doesn't seem to be in your best interest to stay. He doesn't seem all that into you for the long haul. I am not so sure that a poly relationship can be considered a long term serious relationship when its ones first. I would think at his age and stage, being monogamous that he would do better to find a mono woman.
Yup, I think you better start protecting your heart and think about what you will do if he leaves, because it sounds like he is thinking theses things.
Sorry you don't want to hear this and it isn't the answers you want, but I see no other option for either of you