It sounds to me like the three of you are just about at the point where sitting down and hashing out times to see each other on a regular basis is in order. No one has the right to give the other permission, but its time to get real here and come forward with what you all want to see happen. Each person will have to be willing to make themselves vulnerable about their feelings and needs in this in order to go beyond compromise into agreements about time and boundaries.
It does sound lilke you feel guilty. I suggest dumping that feeling if you can and be empathetic instead. It isn't useful in negotiating to feel guilty because people who feel guilty usually make promises and agreements they can't keep. It pepetuates guilt when they find themselves unable to keep them.
The best one can do with that, I have found, if you can't drop the guilt, is to agree as you see fit and if it doesn't work then say so and change it again. Make sure that everyone knows that negotiations are fluid and everything is on trial for now so that there is room to come back to the drawing board and so everyone knows that it is expected.
There should be room for "mistakes" and trying stuff out as long as it doesn't get to a place where someone becomes resentful because they have neglected to speak up when its not working for them. They are responsible for that. No one else is. Good luck
hard work ahead