I don't see guilt as being anything that approaches universal experience for poly folk. I've never felt guilty about anything dealing with my poly experiences.
The only major source of guilt I've had in my life was the teachings of the church of my youth and I shed that dysfunctional baggage a long time ago.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.