polyamorous: in love with more than one (or able to be so)
polygamous: in a relationship with more than one
monoamorous: in love with one person at a time
monogamous: in a relationship with one person at a time
Therefore I see them as kind of unrelated. What I mean is, mono- and poly- amorous are on a different scale than mono- and poly- gamous. Just like you can be a gay man and married to a woman, you can be polyamorous with a single partner (while in love with others) or monoamorous in a relationship with several partners (but only in love with one). In my opinion, because "gamous" comes from marriage, it implies a serious relationship and doesn't include friends with benefits or dating around, for instance.
Therefore I would say that each poly/mono pair is each other's opposite, and that the -gamy and -amory part are neither opposite nor the same, just like a cat isn't the opposite of a tomato.
Polyfidelity means several partners and in a closed relationship. In my opinion it implies both polyamory and polygamy (mostly polygamy as it describes the relationship without talking about the people's orientation inside of it).
Swinging would be monoamory in an open relationship, it isn't polyamory nor polygamy. People consider it nonmonogamous because there is no sexual exclusivity, but in my opinion since there is only one romantic partner, it could count as monogamous.