Is there a chance you could talk to his wife about it, see if she will encourage or..."insist" J date you alone if he is interested as she encouraged your first date? If she is fine with it, maybe he needs to hear it clearly from here again that she is OK with it.
If she knows that you feel weird about the way the dating is going down, that could help change their dynamic for the better. I know that can be a weird conversation to have, but I have found the hardest and weirdest conversations I have had regarding poly relationships with people have often led to the most improvements in the situations I was uncomfortable with.
Happiness will never come to those who fail to appreciate what they already have.