You could tell him you find his communication confusing with mixed messages when he states this to you in this order:
- "Sure, spend as much time up there as you want!"
- "No, I don't want you to go"
- "The decision is up to you".
When you receive mixed messages, rather than forging ahead you could ask him for the bottom line Final Word in order to clarify.
You could tell him you expect him to be a person of his word and say what he means and then mean what he said.
You could tell him you are willing to give him extra time to process things if he needs time to digest and come to the Final Word.
You could tell him that in future you plan to ask him, "Ok, is this the Final Word then?" and you expect him to state "Yes. This is the Final Word on that." And you are going to BELIEVE his Final Word.
And if he chooses to tell you it is the Final Word, and it is ""The decision is up to you" -- then the decision is really up to you!
You might decide to go ahead and he must be prepared for that.
He could not offer you "fake choices" like it is some sort of mind reader test if he doesn't actually mean that he is ok either way and for you to decide yourself. You are not a mind reader. He could just tell you direct what his preferences are in his Final Word so you take that information on board. Does he want his information taken on board or not?
If he gets upset later that his Final Word was actually not true -- he is NOT ok with it either way -- he can deal with it. Because he is in charge of himself and you expect him not to lie about where he's at inside.
You cannot control him or what comes out of his mouth. He does. You cannot magically know things without him telling you. So he has to tell you honestly where his temperature is at with things.
On the flip side -- exactly WHAT did you ask him?
"We'd decided to also go back to M's place for a few hours so we could spend some time together, since it'd been three weeks since we'd last spent any amount of time together. "
If that means sex, say so. For all I know you are "spending time together" playing Scrabble. You could state it more clearly with something like this:
"I plan to see M and Friend for lunch. Sex could be on the table if we go back to his place later even though we have not been seeing each other Sundays and we'd planned to dial it back. Are you ok with this possibility/pace? Do you need me to go slower or just not tell you when sex comes back on the table? I know you had to endure relationship explosion with me recently. I know what I do affects you. Give me feedback, please."
Can't ask for clearer communication from him when you don't offer same. Given that you had recent "relationship explosion" and were planning to dial it back -- depending on what you asked him -- he may have thought you were asking if he was ok with a lunch date and not been prepared for it to be sex too. So now he's mad.
Sort it out.