I'm wondering why you don't do more things alone with him if that's the issue. If you're having a date night, have a one on one date night. It's hard to have any quality alone time when you're in a big group anyway.
Everything will be ok in the end. If it's not ok it's not the end.