Just make sure that you know exactly
where that line is. You just described a very slippery slope. If he can hold your hand, then can he hug you? If a hug is fine, what if you kiss in greeting (you know, just the cheek)? Surely if your husband is alright with kissing on the cheek then he won't be hurt too much by a just a real kiss... and he's not here right now, and maybe he'll never know right? Then the hands start to drift. Etc.
It takes a lot of strength to settle for boundaries, and it sucks to do, but people in our situation have to do it. Just be careful. Broken trust takes years to mend.
I have to say though, I couldn't handle a limitation that gives me so little touching. If my wife can accept that I'm going to have a romantic relationship with another person, than I have to at least be free to hold hands, dance, hug and kiss. Anything further than that requires a talking/arguing session.