I know exactly what fears you have, because I had them 10 years ago when my husband first talked to me about opening our marriage. Luckily for me, this conversation didn't take place after a hurtful incident like yours did. Articulate your fears--be specific. Ask him if he can assure you this won't tear your family apart or embarrass you. You sound open to the idea, but it will take time to quell those insecurities. I didn't take a lover of my own until after I knew we were done having kids and once I did I realized my husband had been right about how it doesn't diminish in any way our love for each other. It's a way to have fun and get some frustration out of your system.
Make a few rules, like no coworkers or mutual friends. Talk about what you really don't want him to do, etc.