I wonder if he sees the fact that you are not into swinging right now as an ultimatum... no love allowed for the other man? No swinging... that kind of thing. Not that that is what you are saying, but I wonder if its how he sees it.
Give this some more time... it sounds like not only you are confused, but he is also. It's hard to get your head around the difference between swinging and poly for some people. Swinging is totally monogamy, hetro based... it isn't surprising that there is a struggle to understand that love is infinite sometimes and uncontrollable. Once that feeling seeps in, its hard to change it and go back. It is possible, but not unless you really want it. It sounds like you don't want so much of the casual thing any more... because there has been a change in you.
Anyone want to be friends on Facebook?
Send me your name via PM