I never did anything wrong
Wrong no.. but you showed you don't trust. Thats tough to overcome in the best of relationships.
Maybe make an effort in the direction of trust. Seek counselling of some kind, or try to have open discussion with him about why you lack trust. Relationships are built on foundations of trust. It doesn't sound like he has given you any reason not to, so trust in that.
Easier said than done but still doable