Annabel put the whole situation very clearly.
What I would do in your position is try to change my perspective. There is a pattern here that you are not satisfied with: you both agreeing to strict boundaries when he has no other interests and him breaking them (or trying to change them) when he has. You can do little to change his part in this, only he can control his own behaviour. So what is it that you can do to make the situation one you are satisfied with?
For my part, I would definitely have an issue with being in the position of having my trust broken over and over again. Agreements have no use if only one party sticks to them. For me, it would be the time to make some personal boundaries around not ending up in that position again. Whether the relationship would survive with those personal boundaries would remain to be seen, but if it comes down to that I find it healthier to protect myself than the relationship.