How long have you been dating each other prior to opening up your relationship.
Has he articulated the reasons why he's not crazy out you dating/banging with in the social circle.? Being outed with extended familiar work......mess up friendships....etc.
Clearly you've set up a hierarchy ......have you thought about the what if's ..if that were to change. Somebody falls madly in love ...finds their soulmate. Their super better perfect soulmate. I'd suggest you both dedicate a few hours and read the different stories here.....get an idea of how things can change and at what speed those changes can happen.