This sounds a lot like the situation I'm currently in, so you're definitely not alone in this. I think the key to all of this is communication and a good understanding of everyone's comfort level.
It sounds like you and your wife have a healthy level of understanding, which is great! Just remember to keep expressing your concerns to her (and to him) and I think everything will be ok.
Best of luck!