My 2 cents as a noob... The way I see it you have three options:
1. End the relationships entirely, no friendships, romances, or acquaintances with either of them. -But this is not something you seem to desire as you care for both persons and want to be with them.
2. Carry on as you have been. -But as you say you hate keeping your intimacy with the wife a secret, so this would mean making a sub choice:
a. Continue your clandestine relationship knowing that you are lying to him about your involvement with his wife.
b. Discontinue your relationship with the wife so that you can move forward with an honest relationship with both of them.
3. Be honest with them about YOUR feelings. Sit them both down (together this time as you have already had separate conversations) and tell them how YOU feel about both of them and that you would like to discuss perusing a relationship with both of them. Then give them time to discuss it together. From there it is up to them as to how they would like to incorporate you into their relationship (or not).