I never worry much about what other people might think. I don't think of my relationships as being anything other than normal and I act accordingly. Should somebody ask who the babe is when I'm not with my wife, I simply tell them--their reaction is their problem.
I suspect that if I treat my relationships as something Oddly Notable, other people will, too. If I treat them as matter-of-fact business-as-usual, then others are likely to accept them as such. If they don't, I don't really care.
So I can be found out with women who aren't my wife. I can be seen holding hands with a woman other than my wife in public and I don't stop to think about whether that might be acceptable--it certainly IS acceptable and I won't tolerate any foolishness that says otherwise. What other people think about my marriage doesn't concern me.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.