To each their own.... responsibility for self.
I am with GalaGirl; everyone involved is an adult, capable of speaking up for themselves and making the choices that are right for them.
It sounds as though a precedent has definitely been set over the past three years, and whether random people call its structure unfair or not doesn't change the fact that you've been investing in this power dynamic for a serious stretch of time.
If you are unhappy, the owness for communication of this lies with you - we have to be our own advocates in our life; you have the ability to speak to your partner, and to his wife (you have her email address, and use it regularly, it would seem). I would encourage you to find your voice and speak your feelings out loud. Perhaps boundaries that worked in the beginning haven't had the opportunity to be renegotiated as of yet, and this is the time that it's meant to happen.
Maybe there isn't room for renegotiation. Maybe you'll still want to be with him despite that. It is your life, and these are your choices - finding peace with them might be hard if you want something different than you have though. Hugs, doesn't sound easy.