Greetings and welcome.
First question : Do you have any actual confirmation that his wife knows about you ?
As for the 'agreement'..... Right now I would be a very cheerful puppy to get to see the someone I desire, every two weeks. :P
But that is me, and not you. The point of that cheeky comment is; that agreements need to always be open to renegotiation. The situation and timing works for him, but does not work for you. Based on different lifestyles, it is definitely a area that needs to be kept 'open' not 'ruled'.
Since it doesn`t work for you, it is not a good agreement.