My gut reaction is that while logically, you're comfortable with the concept of being in a polyamorous relationship, there is still nagging doubts in the back of your head. Most likely remnants of being conditioned into thinking that this is wrong and that anything other than 100% monogamy is thus him cheating on you and leaving you. It's natural---these are feelings just about anyone will experience as they are first dipping their toes into the non-monogamous waters.
I would say give it some time and ease your way into this. Developing some local poly friends and see how they live their lives in happy polyamorous relationships could certainly be beneficial.