Is this just about the sexual aspect? Perhaps I am misreading what you wrote, but it seems that way to me. If so, you may find more answers on a site devoted to swinging, since that is primarily about the sex.
Do you have a desire or capability to fall in love with another person? While sex is often a part of polyamory, it's not the only part.
"Listen, or your tongue will make you deaf." - Native American Proverb