Polyamory is not about sex. You can have sex with people you don't love and you can love people you don't have sex with.
Poly is about the love of many or multiple people. For me, it also has connotations about the possibilties within relationships, but that's more about my personal journey than about the term itself.
A person is polyamorous. A single, celebate person can be polyamorous.