I wrote you privately, but will write here too.
I have been in a very similar situation. Being in contact with him will not be very popular (at all). Lots of folks really look down on cheaters and the people who love them.
I do understand that it is hard to only meet poly people - to steer the heart - I think in these situations, you can find yourself in unethical situations.
I think you just have to be honest with yourself, and decide. This is definitely an emotional affair - if he is relying on you to talk about his marriage. There is definitely some inappropriate closeness. I do not agree though that your presence will be breaking the marriage. To me, it's all on him. He was seeking out someone and the problems in the marriage are why this happened.
But what do you want? Do you want someone who may lie? Who you probably won't have sex with? Who you will get little support and lot of criticism for having in your life?
I would, at this point, let him go - you barely know him, and the conversations may be habit instead of attachment. I say this with NO judgement. I decided to stay in my emotional affair, and things have worked out into a somewhat platonic support system, but the relationship is static and keeps me from getting one that can be open and sexual. But that's the unpopular decision and I can't really justify it.