2 very important comments
How much experience did this couple have with secondaries before you came into their lives? Is the wife experienced at secondaries falling in love with her husband? To me you made 2 very important comments that need to be explored with HIM:
1. I think bf feels offense that I am not a fan of hers?
2. At times, I almost feel like it is "you had better be friends with her, or she will not allow this to continue".
I would be open and share those concerns with him. Maybe they are coming from a real place that needs to be dealt with.
The fact that your feelings for him are increasing balanced with no desire for a relationship with her could be trouble.
My relationship started similarly although she and I were friends and she was actually the one that invited me in. I have limited experience outside of this couple, but after 5 years I can tell you this. As my relationship got deeper with her husband she became more defensive and needy.