It doesn't sound like you'ed been talking to him very much about what you think a relationship would be. It sounds strange to me and I would have a very hard time dating someone who needed to function that way. But have you talked with him? Is this a "always and forever" thing or just a "while we both get used to you" thing. You said you know that his wife was okay with the first alone date but have you ever talked with her about it?
I don't know if I'm reading this wrong but to me it sounds like you weren't communicating with either of them about your expectations and they weren't communicating with you so when you hit the (inevitable) wall you got angry at him and told him it couldn't possibly work without ever sitting down and talking about what all three of you need/want from this relationship. There are so many ways to do non-mongamy and it sounds like all three of you were assuming that the way it was in your head would be the way it worked on all sides and that clearly isn't true. If you like him I wouldn't give up until you figure out what there real limitations, boundaries and desires are. Give them a chance to hear your feelings and be willing to work with them to find something that works for all of you. Maybe that something doesn't exist but if you really like someone it's well worth a shot.