Your hubby`s response is him verbalizing his fears. It`s like someone who is scared of plane ride`s chanting 'I am gonna die !'
I think you owe it to both of yourselves to at least 'try' this upcoming weekend. You will both have a clearer idea of what you can/cant handle after the first sleepover is done and over with.
You plan on keeping in contact with hubby and coddling his nerves a bit, but not to much. Good for you.
Don`t get to angry at him. I DO understand the 'fairness' issue that can come up, but you need to lose the battle to win the war,..so to speak. For him, it is much like him having a good hand,(sexual relations and time with others) but you hold the trump card(feelings).