Any way to correct that so that you and he are spending more alone time together? Can the time be set aside? Can special arrangements be made? Special date nights or what have you?
I think as long as you guys are sensitive to each other's feelings and needs, and of course, communicative about them, you will do alright.
Love means never having to say, "Put down that meat cleaver!"