Your fears of falling in love with new partners and out of love with current ones are common. They speak to a wholly monogamous view, however, as the underlying assumption is one of romantic love being a zero-sum game, that you only have X amount of love and if you give that to one person, you have none for anybody else.
I suspect you'll be quite relieved to find out that love is not a zero-sum game--that you're able to love more than one person at a time without losing any existing relationships. It's a whole 'nother world when that realization hits.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.