Would having a set schedule work for you? Say certain days/times are aet aside for each of you thatway you ccan't feel resentful for the time because it's what you agreed to.
Also put yourself in his shoes. When you have a partner wouldn't you want them to be able to come to your home?
Sue, hinge in a vee with Nate (Polysexual) and Sam (monogamous)