What is he not getting?
If he is a "second" then it makes sense that he would feel lesser than your "primary" relationship. I don't see why he wouldn't, unless that is a position which naturally suits him.
If O wants a relationship with you, one with love, concern, commitment, adoration, and time then being in a secondary (lesser) relationship would naturally be uncomfortable. Perhaps a monogamous relationship, or a non-hierarchical polyamorous relationship is more his style.
I would want to understand more about what he was after, and what it is that you can't offer. Maybe he *can* get what he feels like he needs from you in the current structure, and he is just over-estimating how "primary" your husband needs to be.