Remember, you can do poly without doing open. If casual sex makes you uncomfortable, then you can agree to not have casual sex. That would put all sex within the bounds of serious relationships, of which you can have many.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.