His (and her) fantasy of forming a poly tribe doesn't have to become your fantasy, just because you're involved with him. You certainly have a right to your own autonomy and don't have to move in with them, date them, or even be friends with them if you don't want to. If you need a certain amount of distance, express that need to him. A polite and friendly acknowledgment of his wife and family should be sufficient if you do not wish to invest in that aspect of his life. If that isn't good enough for him and them, and he becomes insistent, then you both will have choices to make about whether to continue or not, and where to make compromises and set boundaries.
Don't ever feel that you should set aside your needs or put yourself in a situation that makes you feel like you are at odds with your convictions, or with what would make a relationship satisfying to you. A little discomfort is okay - we all learn and grow from that, but you shouldn't have to twist yourself like a pretzel into something you're not just because he wants certain things from a relationship. You don't have to sacrifice your privacy if that is important to you, and you shouldn't have to defend yourself for needing that either.