I don`t think you need to know what you want as far as 'poly/swinging/open' before going in. Sometimes you just go with the flow, and sometimes you don`t know until you try different things.
I agree with nycindie about the passive-aggressive, wishy-washy communication. This is how shit happens. You have to say exactly what you want, feel, and desire. Code words and invisible markers, are just not going to do the trick.
My back is up, reading about your boyfriend a wee bit. So,..you go through a abortion,..and your sex drive went down. This is solved by,....him going elsewhere ? No wonder you are having issues.
Then he had a hissy-fit saying there was to much drama ? That strikes me very much like a child having a temper-tantrum. He might of just said it in anger, and that can happen.
Also,..'freaking out' happens. To a MAJORITY of people. There is no award given to people who STFU and suffer through their hurts.
You two need to learn to be more honest and direct with each other. Much like nycindie said,..this is just blunt-speak, take it for what it is worth. Good luck.