Deep breaths!!! One thing that I always remembered when trying to find someone for my wife and I was that my marriage always came first. Even if it was someone that was more for me, she still had to be comfy with that person. I gotta be honest, I would have nixed a person who said they didn't care about how my wife felt. That just wasn't an option for me.
I know you're caught up in a lot of NRE, even if things are purely NSA, and it's tough to give that up so you can make hubby feel better but it may be something that you need to do now so you can proceed with this lifestyle down the road. Know what I mean?
Live life to the fullest 'cause you never know if you're gonna wake up tomorrow!