It sound like if he were to decide to withdraw the relationship or just the sexual aspects that it wouldn't leave a huge hole in you schedule. It sounds like you divide your time very equally. Perhaps if it were to happen 1 or all could step up and fill in that gap ...perhaps rotate that Sunday time slot.
The options sound like trying to preempt the pain of being dumped or dropped.
In a poly mindset isnt he just gravitating toward something you can't give him.....his need for more than one day a week....his need for a gf. Maybe reframe it like that .....you aren't fulfilling all his needs ....you can't by nature of your other relationships and schedule.....and he's made the calculation on the loss vs gain ...for him. What about compersion?
I'd enjoy the time you have and be happy.