Yeah, I'm not sure where the line is with kinky or not kinky. I see it as meaning a fetish or proclivity that's outside of the "average" sex.
But I don't think the "If doing the same sex act with one person is NOT kinky, why is doing that same sex act with two people suddenly now kinky?" argument holds up. The fact that you're doing it with two people is what's kinky for some people, I'm sure. I mean, you can use that argument with pretty much anything:
"If doing the same sex act while not tied up is NOT kinky, why is doing that same sex act while tied up suddenly now kinky?"
"If doing the same sex act while not wearing a diaper is NOT kinky, why is doing that same sex act while wearing a diaper suddenly now kinky?"
You could even use that argument with things that people find repulsive and morally wrong:
"If doing the same sex act with a human is NOT kinky, why is doing that same sex act with an animal suddenly now kinky?"
"If doing the same sex act with a living human is NOT kinky, why is doing that same sex act with a corpse suddenly now kinky?"
Basically, you can't take the part that people think of as kinky, weird, a fetish or even immoral, disgusting, etc, and say "without this one part it's fine, so it should be fine with it".
Now, personally, I think of threesomes as kinky. I think of pegging as kinky. I think of fisting as kinky. But I also think of them as normal, too. To me kinky only means that it's the type of act you don't assume people do and you don't mention casually in a conversation with your boss. I believe almost everyone has some level of kinkiness, and I think the word "vanilla" is meaningless out of context, as everyone is vanilla for some things.
I do agree polyamory and BDSM are completely different. And I don't think polyamory is kinky because it's not an act. The sex involved might be kinky or not, depending on the accessories, acts and people involved. The relationship itself isn't.