There are several red flags I see in this situation, but the biggest and most problematic is that he is maintaining a secret affair with a married woman. The two of them are now involving you in keeping this secret. This is a blatant violation of trust -- for everyone involved --, and is likely to explode in all of your faces at some point. And even if it never does explode, as I predict, it's no way to be treating this other man, the one who isn't in on the secret.
That he, your partner, is willing to engage in such a secret affair should be seen as a red flag by yourself, CharlotteCorday. If he has been dishonest with you and remains dishonest with the one out of the loop, that's not a sign of good things. It's a bad sign. It doesn't look good at all -- especially for a person such as yourself who places a high value on honesty.
How much is he really loving you? He brought you into a mess and expects you to go along with it. Right? That's a power imbalance besides the other issues.
I think you deserve better.