In my experience I feel the same way your husband does. I have told my husband on many occasions if he wants to leave me that is fine. He tells me he wants to stay with me and that leaving is not an issue I should even worry about.
In my mind I can only see the things from my perspective. In my mind my husband will do what is normal he will fall in love with someone else and not want to be with me.
Deep down I know this is not true and not what being Poly is all about but it is there. Even though he states he won't leave how can one predict that much future??? I know things will be different in time though.
Maybe things will change between you as time goes on. Maybe he will see that you have a limitless amount of love to share with both of them.
I think it is just hard to go against the norm so to speak.