He offered to include his wife in the conversation? Oh, and as for the tongue-in-cheek comment, usually there is a grain of truth to every joke. While you two are "safe by distance" in his words, there is such a thing as emotional cheating which may very well be what she feels like is happening in his relationship with you. Emotional cheating is generally a difficult topic in relationship because often men in particular (I'm sure there are women like this too) that don't view it as cheating until something physical has happened, while it is still an emotional betrayal to hide things (like loving someone else) from a SO. Hell, as a poly/open person I'd be pissed if my husband hid feelings for someone else, why wouldn't I be if I was mono and expected him to not develop them?
Why don't you just tell him that you know his marriage has some issues, and until those are resolved or at least being worked on, you don't feel comfortable going down that road. Would that help encourage him to get the help he needs for his relationship? Do you know much about his wife? Would she possibly comfortable with him having a LDR with you, once their marriage is solid?