I'm going to try and repeat back to you from FV's perspective what you have shared (with some creative deduction,-get out your grain of salt)
For six months, FV was repressing sexual desire for you out of respect for the relationship between you and DW. Six months of denying himself while you carried on with your own sex life with DW.
Finally the three of you work out a space where you and FV can be intimate- and it is nerve-wracking for FV. He is used to being restricted by your and DW's rules. You are in an unfamilar place (hotel), there's the awkwardness of the first time, and it turns out that no, really you can't be intimate.
Enter new woman that doesn't impose restrictions at all, it seems. He finds release with her. With you he is still operating in terms of rules and restrictions so he focuses on that in defense because he knows this will be difficult for you, but still has the balls to tell you.
He is being honest, maybe not doing exactly what you want him to do, but what he feels he needs to do for himself. Remember that you have been doing that exact same thing for six months and good for you for waiting until you and DW were ready. I hope you all keep doing it.
He wants to be in your life, he said. If you want that too, let him be. you can work out the kinks, but don't give up if you have the emotion.
My perspective is surely incomplete, but this is how I read your post. I hope you find light and warmth for yourself again soon.
"Rocks will open and make a way for the lover."
~Hazrat Inayat Khan
I love Catfish and Charlie.