I've noticed this dynamic before and it absolutely stung at first since, at least for me, my initial reactions in becoming poly was to internalize any disparity I perceived. It seemed there were subjects that got my husband hot with others that he was uncomfortable to try with me because he worried about my reactions more than with someone else. With me, there is more to lose should this relationship go south so of course he is going to stress over that possibility more with me than any other partner.
I don't know what has happened between you and your partner in the last 6 years or what pain those years manifested. It might help to explain whats up with that. But for us it was something we battled over repeatedly. It was when it got to its ugliest that we made our biggest break through. He began to take the risk of losing me by bringing me closer to his sexual wants rather than shielding me from them. And I really dug deep to not turn away from him out of the hurt that he had been sharing those parts of himself with others and not me. Its like a whole new relationship now complete with the NRE I thought people only got with new connections.