I'm guessing quite a bit here but it sounds like your husband really liked 'sharing' you with other men, and then hearing about it later, leading to the great sex between you two. And he liked being with another woman while you were with that's woman's husband. It's a pretty common fantasy for many men (and women).
Anyway, it reads like your idea of what an open marriage is might be different from your husband's. He seemed to think things would continue as before while you believed that the two of you would see people separately. Neither scenario is 'wrong' as long as both of you understand it and agree what an open marriage means.
Time for long, honest talks!