The Wrong Question
This isn't so much about polyamorous vs monogamous relationships, but about closed vs open relationships. A good portion of polyamorous relationships are closed, and some monogamous relationships could be open.
Clearly if two people are just naturally monogamous, but there are no rules in the relationship saying that they have to be, you can't call that abusive or slavery. Presuming that everyone is secretly polyamorous is without statistical evidence. It's quite possible if everyone had open relationships, some would still be monogamous.
So the question should be: Are closed relationships akin to slavery? And are they abusive?
To answer this question we have to look at what a closed relationship is. There are two qualities.
1) A closed relationship demands that one person or a group of people have a right over what another person does with his or her body. If they didn't have that right, if people just had the liberty to do what they wanted with their own bodies, then it would be an open relationship.
Calling this slavery will naturally cause reactionary comments, because the word slavery has such negative connotations. However simply by the fact that you are exerting control over another person's body is akin to slavery, though it's more like wage slavery than the kind of slavery based on physical bondage. Wage slavery is a mutual agreement, which in many cases people can opt out of. You aren't being forced, so much coerced to stay in such an agreement. In the case of wage slavery with the loss of wages, in the case of a closed relationship the loss of a relationship.
Is this abusive? We have to look at the second defining feature of a closed relationship.
2) A closed relationship has to be contrary to some of the urges of someone or everyone in it. If this was not the case, there would be no purpose of having a closed relationship with rules. Such rules would be unnecessary. So this might seem abusive.
However, if we can go back to my previous example, participating in wage slavery is a necessity for many people to survive in our society. Is it abusive? Not as much as starving or living on the street. Things are relative, and in a society like ours, most people have to make pragmatic agreements in order to survive and attain some measure of happiness. Whether they are ideal or not. So wage slavery might be abusive but not as abusive as the alternative, at least for a lot of people.
In our society, people are taught that some jealousy is a right and is acceptable. In fact it is ingrained in people's mind that you should be jealous in certain situations. There are cultural pressures to be jealous. In this sense, while people might have urges to be with people outside of their relationships (whether it is romantic or sexual urges), they would prefer to suppress those urges because they would be horribly jealous if the other person or persons did the same thing.
I.e. since I can't have a relationship in which I can be with whoever I want, but the other person can only be with me. I would prefer a closed relationship in which we can only be with each other. My jealousy outweighs my desire for other people. (This is putting it simply, certainly more noble people would think if they were going to force someone to just be with them, they have responsibility to reciprocate.)
A closed relationship, in this sense, is a preferable -- if not ideal -- compromise for the people or some of the people in it. Is it abusive? Sure, but it's less abusive than the alternative for a lot of people. As our society has trained us to object to non-abusive relationships.
Somebody here mentioned BDSM, and I think it's an apt comparison. In a healthy society, would you get off on bondage and control? Submission and domination? Let alone masochism and sadomasochism? It's very conservative and authoritarian, it goes directly against any kind of libertarian ideal of free mutual support of one another. But we live in a fucked up society. And people's sexual predisposition is often developed at an early age. And if that is how people get off? Is it really a bad thing if everyone agrees?
If agreements are free and fair, and to the benefit of everyone involved, can you condemn it? It might be abusive and yet still a positive agreement for everyone involved. Once again, because the society, in which we live, makes non-abusive relationships impossible.
It's only becomes an unhealthy abusive relationship, if it isn't for the benefit of everyone involved. The most common example of this would be non-jealous people stuck in a closed relationships. In this case, someone is likely being restricted in the closed relationship, but gaining nothing from the fact that the relationship is closed. Because if they don't care what the other person or people in the relationship do, but have romantic and/or sexual urges for other people themselves, then the closed relationship is harming that person on some level. Note I say relationship, they aren't really being abused by the other people necessarily, but the situation.
However, because we live in a society that is so supportive of closed relationships, if a non-jealous polyamorous person can be in a closed relationship without going insane or being perpetually hurt... you can't really say they shouldn't be in a closed relationship. While they might prefer being in an open relationship to a closed one, they might prefer being in a closed relationship rather than being alone -- if you demand open relationships, your options for partners is far more limited. This is once again unhealthy but probably preferable for some, and it is just dickish to harp on them for being enslaved or abused. People can make their own decisions here.
It's like if a person lived in Kansas. Say, in turn, that it is a particularly conservative town in Kansas, and they don't have the means to escape that town. Say this person could only be sexually attracted to people of the same sex, but could be romantically attracted to people of the opposite sex. Would you judge them if they chose to be in a straight relationship?
We have to accept that in our society some relationships, while philosophically offensive, are more acceptable than they would be in a healthy free society. Closed relationships are one example of this. Condemning people for participating in closed relationships, when they live in a closed society, is tantamount to cruelty.