Hi, I liked this story. I really relate to it. I'm very new to the concept of polyamory and falling in love with it already.
I think the issue here is the difference in the way men and women relate to sex.
At least from my experience, women are much more emotionally attached to their sex partners, where as relating from my own experiences, I'm happy to meet a woman, have sex with her immediately without exchanging words and then never see her again!
So maybe what your partner is worried about is the possibility that your being with other men diminishes the amount that you require his emotional connection as you could be getting it from others. Also the perceived power imbalance in the relationship, where he may feel insecure about you having a more satisfying sex life than he is, so you may outstrip him emotionally in that regard, with the prospect of the relationship fall apart, or at least be less fulfilling as a result of it.
I found this article very satisfying to read:
I identified with it, so much, it really made sense to me. Maybe you would find it helpfull