I identify strongly as poly.
But, my husband doesn't.
However, he is capable of loving more than one and is interested in having another full-time partner. He just doesn't feel comfortable with the label so much.
Which sounds somewhat similar to what you are saying. I think it's perfectly ok and reasonable to feel that the label doesn't "quite fit me" and so not claim it.
For the Pride Parade this year, we had a float, it was "I am JUST me". Sometimes we get so caught up in finding our "label" that we forget, the most important label we have is ME.
It's great that you are working on yourself-that's important. Try to be understanding that he now has difficult emotions to work through too-trust issues, based upon your previous actions. That also happens to all of us, there's an ebb and flow. It's part of life-but it's important that you realize it, so you can avoid getting caught in the "wtf is your problem NOW?!?" attitude. His problem is that he has to witness a significant enough change in you that he can regain trust that what has happened (several times) won't happen again-AND he has to gain enough momentum to want to risk it again. That's a tough step.