You said "he s also married to a woman who is unaware of our/their relationship."
We have a specific rule about this. We do not have ANY relations with someone who'd SO doesn't know about or would not approve of. This almost always is found out and leads to undo drama in your life. How can you trust this person if he is willing to cheat on his own wife?