It doesn't sound like an issue where you dislike D's personality, if I am understanding correctly.
So to the physical... I am female. I can't speak for all females, but for me whether or not I find someone sexually attractive has to do with personality, attitude, intelligence. Oh sure, I notice pretty people - males and females alike - but they are like art. I have had the experience of finding a very average guy, and even somewhat on the slightly repulsive side, become ever more attractive once I see what's inside his head. I have had it work the other way too - somebody that had physical beauty becoming markedly unattractive when I got to know him. So your wife may be seeing him in a completely different way than you do. If you just paid attention to D's personality, what would you see?
Now, the bottom line is I know D does not fit your fantasy of watching some hot dude, bang your wife. Separate that out of the equation too. Them remember, you aren't having sex with him; your wife is. And in this situation, the only important thing is that she does see something in him. You don't have to.