Do you continue because this is the way poly relatioships are? I'd say who cares how they are .....what do you want. I didn't read the other posters so this might be redundant ...but it looks like your good enough when things line up for him and his wife. The big IF. If I'm in town ...( in your town ) ...if I got time ....if the weathers not too bad ...if my wife give's me my balls ( kept in her purse ) ....if I don't have a heavy meal or one containing MSG.
Id find someone a tad more invested or at least closer to what you want out of the relationship.
How did you find this guy ?