That makes it alot tougher. Perhaps its for the best. Let the wife find her own partner. If there is no connection between them, you cant force it. Didn't you say that you were all doing the swing thing before. Maybe the spouses have sexual attraction and nothing more. I think you need to speak with D and tell her your feelings. Then you will know if you should proceed or not. I agree with your wife, slow down and let it develop. Best of luck!
I would rather be hurt by your honest than pleasured by you lies!