If he really wants a relationship with you, it has to be either with her along or in a quad... geez. It sure does sound like his wife is super insecure and controlling if she has to be there and have a say in every little thing. Are you friendly enough with her to talk about it and find out? Even though you've pretty much written him off as a lover, I'd be curious.
He sounds like someone who thinks that's how poly has to be. Like, he's not even hearing you because what you're saying goes against his ideas of poly. Or he's just whipped.