I'd say that it seems like at least part of him has grown attached to the idea of something longer term with you. It might be worth talking about to figure out where his feelings lie in that regard, and where yours do. What about his wife? Is she involved in any conversations of this nature? What is their agreement (if they have an explicit agreement) about relationships outside their own? Is there any possibility that you might be able to have something more permanent with him without him ending his marriage?
Having been on the other side of something like this (as the wife of a man with a girlfriend, until recently...) I can say that it may be threatening to her if these feelings / ideas of something more long term between her husband and you are in contradiction to agreements or expectations they might have. After you talk about it with him, assuming you're interested in exploring something longer term, or just getting on the same page about this, I'd talk with her, also.