Welcome to the forum
Sounds tough- almost like he wants to want a poly life, but when confronted with it, cannot deal.
I wonder what he has said to you- i.e.- bringing it up- what conversations took place after the first encounter and before and after the subsequent ones.
Does he want the arrangement you describe? You have two lovers and he remains monogamous with you?
What of your husband and lover's friendship? Does it still exist or is their relationship purely hostile at this point?
I look forward to hearing more about your situation and getting to know you.
"Rocks will open and make a way for the lover."
~Hazrat Inayat Khan
I love Catfish and Charlie.