Yes, he was affectionate to me.
I doubt he's staying in this relationship to avoid hurting you. It's sounding more like he has feelings, but is trying to deny them, and can't make a break from you either.
It think his holding back is an attempt to control his emotions. i.e. "If I hold back my actions, I won't continue to develop deeper feelings for her." We both know this won't work.
So what do you do? It's hard to say. I'm always one for laying things out, and going from there. It sounds like doing so with this gent might scare him away? Perhaps wait for him. Don't withdraw the amount of love you're showing, but don't push for more. Let him come to terms with what he's dealing with, while showing him you're still there and are prepared to wait for him.
If that's how you feel.