I hate to say it, but it sounds like he's been misleading you, making it sounds like they were more distant than they were. Not answering a direct question is a pretty bad sign. Perhaps he's also been misleading her about his relationship with you, in various ways. It also sounds to me like maybe SHE made the decision to leave HIM, which means he hasn't *chosen* to be with you at all, he just doesn't have the option to be with her in the same way anymore. I don't know either of you, and maybe I'm entirely wrong, but there are SO many red flags here. I'm afraid you may have walked back in to a very bad situation.
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.