I don't have experience with this but I can say that it's best you don't ask him if you should give your lover up. Generally the answer and feeling behind it is 'YES'. His insecurities are already loud and questions like that will give him the impression that he has control over what you do, when he really doesn't.
I think you can definitely try negotiating and letting him know that his feelings matter, and investigate why he might be feeling the way he does. Since it hasn't been the case until now it's important to find out what could be causing the sudden decline in self-image.