Firstly, I think you were very right to say no, both because of the agreements you'd made, and because if you'd given them more time so they could have sex, you may have resented it when you went out to dinner and felt badly, or it would be awkward for everybody. From your followups to the situation it sounds like you were ideal in your behavior and caring for both of them.
Secondly, as it's gone on, I think you are putting her or them ahead of yourself, don't bother being grumpy about the time they've gotten together this last week, but don't be an active participant in giving TOO much, and worrying about her needs, or his needs, tell them, and let them be their own advocates. From what you say it seems like you are sad you are advocating for them, but don't feel either of them are advocating for YOU.
You need to be your own advocate. Try not to worry about them, think about want you want and need. 3x a week where he is home? 2 date nights where you focus your attention on each other and phones are off? Him to program his phone to remind him to initiate sex every Thursday night with you?
It sounds like you are both doing an awesome job. I envy that he realized he'd not done well by asking for that "favor" and didn't try to squirm out of it, you have a great partner there. I'd also think you're a great metamour. Just make sure you don't let yourself say yes to things when you want to say no, it can be a slippery slope, but you sound like you are trying hard to keep yourself honest.
Happiness will never come to those who fail to appreciate what they already have.