It sounds like you and your hubby are not really ready for an open relationship. You have so much to work on and heal in your own relationship before getting others involved. Neither of you trust the other and it looks like the communications has broken down between to two of you. Those are, in my book, the two most important ingredients to having an open/poly relationship.
Is your fear of him leaving you unfounded? I hate to say it, but no. From what you've said here, your relationship sounds like it has just been sitting and waiting to fall apart. But I put up my disclaimer of not knowing all of the details.
You two need to sit and hash EVERYTHING out, perhaps even find a poly friendly counselor to help you through some of these issues.
I wish you luck.
Life is about the journey and not the destination,
so what better way to know life
than to wander all the roads and paths set before you.