This sounds like a hard place to be. I think your concerns are valid but you have to ask yourself; what can you really do? If you refuse to let your wife see him then you will be right back with the problem you started with. At least it seems that way to me.
I sincerely hope that everything works out.