time heals .....
You will probably have to give him time... from your NRE perspective; lots of time.
My guess is he needs to assimililate his new "open" relationship and even if he knows intellectually that his wife isn't leaving him, his emotions tell him differently. Plus now he also has the hurt of thinking he's not good enough/man enough for his wife because she also wants to be with you. (Again intellectually he may know thats false but feelings say otherwise.)
He doesn't sound as if he's as confident as you, your wife or your wife's best friend; this is probably crushing to him! Try to put yourself in his much less confident shoes; feel it.
With the passage of time as you and you "gf" adhere to the ground rules, he'll find more and more confidence in the love his wife has for him and that she's not going to leave him.
You've changed the whole playing field; PATIENCE & understanding must be your motto.
I know how hard patience is; I am/is you! It's not easy but it will be worth it.