I think the bigger difference is the removal of a gender bias within the context of a relationship definition. Polyandry and polygamy assign a role to the genders which implies a lack of choice.
Polyamory does not. There are no gender roles, in fact no set rules to design. There are endless configurations possible, straight, LGQBT etc. Its a big step forward and far easier to accept and explain than the gender restricted variations. You still find polygamy in the world of poly...but it doesn't need to be defined that way.
Thats my take anyways.