I can understand why it would be hard for you not to be able to hold hands. It's hard for me to stop myself from doing what I would normally not think about twice! But I think your partner's comfort should come first. Of course, if they don't want any touching in public, chances are you were aware of that pretty early on, so if that's a deal-breaker it's easy to spot.
I think I'm probably like you, being unable to touch at all would be too hard for me. If I can't kiss my partner, at least I can hold his hand, caress it or squeeze in gently, or kiss it, etc, depending on what he's fine with or not. I need some level of physical contact.