I'm glad talking it out has helped you some. You seem like you have a plan and your boundaries set -- you are prepared to let him deal with his stuff.
It makes me feel that I'm being harsh towards him by not agreeing to his request
What's your plan for your OWN emotional management? It's not "harsh" to say "no" to people -- sometimes it is necessary.
I should have probably made it clearer in previous posts that he is very intelligent and very capable, he doesn't need 'care', he just needs help with issues around emotional intelligence/understanding people's feelings and motives.
We all need care. From ourselves, from others -- in varying degrees. Your Asperger spouse needing help with this?
That's some level of care to me. And I see you trying to find a balance for providing help without getting sucked into things that you think are "meh" and learning to say "No, thanks" and being ok with it.
Him taking up with this woman in this context is "meh" to you -- she's clingy/needy, he's not prepared to dom well, etc. You rather stay home than attend this potential scene -- so stay home.
Let it be what it will be.