Welcome to the forum!
It certainly sounds like you are doing a lot of things very right - you are in contact with your partner's lover and have built up a very functioning relationship.
You say you aren't afraid of him leaving her for you, and you say that your relationship with him has got even stronger. So what are your fears?
"Listen, or your tongue will make you deaf." - Native American Proverb