Might be a good idea to suggest he join the forum and join the discussion.
If you were open from the beginning and he says that the rules have changed, it sounds like you have had different views about what "open" really meant. I realize that your relationships were swinging, but it's not clear that you had a "just recreational sex" rule in your relationship.
His anger and lack of consideration of your feelings is very worrisome. It does not bode well for your future together. He does not seem to be dealing well with the conflicting needs of you two. That stress is coming out in bad ways.
Unless of course, this is a long-standing pattern of behavior in terms of his ability to deal with difficult situations and his level of consideration for your feelings. If that's the case, either get in couples counseling to try to break the dynamic or plot your own path going forward.
Male, Straight, Poly
“Instead of getting better and better at avoiding, learn to accept the present moment as if you had invited it. And work with it instead of against it. And making it your ally rather than your enemy.”