I understand your problem I too have a long distance relationship (though not that long of a distance) with someone. I had first met her after she was married and though we have never physically met we do share a deep emotional connection with each other. I have spoken with her husband many times about our relationship and he is understanding in the fact that the two of us can share our connection together, however not excluding him.
You might want to talk with both of them together and discuss the concerns that you have. Make known to all of you what are the intentions of all parties involved, you, him, and her and work it out. this however might cause a problem IF she does not know about you (and I say IF) or if she thinks of you as more of a weekend friend or the FWB type. Everyone needs to be comfortable with each other, as is with any relationship.
Jeez if I followed my own reasoning at first I wouldn't be where I am today.