You've mentioned a couple of times already that you are fulfilling her in a way he can't ... it comes off a little rude, to me. I wonder if he is picking up on this vibe from you? That would certainly make me not want to spend time with a metamour, if they were smug about being able to offer something (regardless of its nature) to my partner that I couldn't.
And remember, it doesn't matter if you mean it this way or not. He may just be interpreting you this way, as I have done.