Guys will sometimes withdraw from a relationship because they are approaching a point of truth, that they are in the closet about something and do not have the courage to come out. I am not saying that this is going on, but this comes to mind. Issues such as another lover, bisexuality, crossdressing, or various fetishes can drive (more so guys than women I think) someone so very deep inside. Of course you have no obligation to deal with such issues. Women frequently see the withholding of such deep secrets as a violation of trust, and find that breech of trust more damaging than the issue itself.
Sure, we can say to the guy, "sack up!" But some guys have secrets so deep that it is easier--and more tragic--to carry them to the grave. Obviously, some of these people can benefit from therapy.
Where does this leave you? That fact that you are in this poly forum suggests that you are probably open to some form of polyamory. (Dossie Easton would suggest that a secrete affair is still poly, but rather an unethical poly relationship.)
If this guy really is that important to you (and this is when falling in love can really suck!) ask yourself what you would be willing to bear, all the while continuing to search for the issue. If not, I'd suggest that you start weaning yourself from your emotional attachment, then get out.