You are smart and sensitive to understand that the the comfort levels of everyone have to be respected, even though you were not able to do so. And indeed, it seems so trite and innocent the hand holding. But nevertheless it was a betrayal of trust. It is so easy for those of us who betray to justify it, explain it, and minimize its significance. But for those who are betrayed, the foundation of their world disappears.
I think that you need to do what your wife needs to heal the betrayal and get the trust back. Hope this helps!