In some feminist literature I've read - thinking of Karen Warren here - there is a distinction between hierarchy and domination; in combination with dualism, they make up the substance of structures of oppression.
Dualism is just the notion that natural kinds can be divided neatly into two groups: male and female, white and black, human and non-human, etc.
Hierarchy is just the notion that something is "higher" than another thing on some particular scale . . . though the choice of scale may be more or less arbitrary. I'm taller than my partner, but she's a better dancer than I am, etc. Humans have more versatile cognitive abilities than non-humans, etc.
On its own, Warren points out, hierarchy is not necessarily pernicious.
Domination is the added notion that whatever is higher has the right to control or dominate whatever is lower in a particular hierarchy.
So, traditional patriarchy combines the presumed dualism of gender with the presumed hierarchy that pretends males are somehow more perfect or more similar to God than are females - Aristotle has a lot to answer for on that particular point - and with the logic of domination to give rise to structures of oppression: males are higher than females and so have the right do dominate them.
Perhaps the same could be said of the primary/secondary distinction. Some reject the dualism outright. Some accept the dualism, and even some version of the hierarchy, but defined in a way that does not join it to the logic of domination. Some . . . well, you get the idea.
Of course, adopting Warren's language in the context of polyamory would be problematic, since some who practice polyamory also practice various D/s dynamics as a matter of deliberate choice. So, using "domination" as a pejorative might be, um, controversial.