While being impromptu is nice, it can't always happen. You may have to change how this looks to you. Maybe a surprise date for just the two of you. You might have to give up on never having the ability to be spontaneous at home and just schedule time.
Maybe you need to talk to NM and ask her to randomly surprise you with time. That type of thing
This really isn't much different than having roomates. Its just the way things have to be. My wife and I live with a close friend and we battle with this constantly. We have to get creative sometimes.