That's where the fine art of negotiation comes into play. The greatest outcome possible from negotiation is that everybody walks away feeling good about getting what they need.
You've already stated that your freedom to do as you please is more important to you than any other person, so I'm not certain that it really matters whether you negotiate much in relationships or not. You've already established that no relationship is anywhere near as valuable to you as it may be to the other person(s) involved.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.