The way I see it, poly is about emotion first, so I don't think it requires being sexual or wanting to be sexual with several people. It is the most common case, yes, but not the only one. If that's how you are, you could define yourself as polyamorous and monosexual, for instance (asexual if you're not monosexual either).
@idealist, asexual people aren't attracted sexually to anyone. It's an orientation like heterosexual, homosexual, bisexual, pansexual, etc. The prefix "a" is a privative, meaning sexually attracted to nobody.
They can still be heteroromantic, homoromantic, biromantic, describing who they might fall in love with. The sexual attraction simply doesn't exist. It's not about the choice of resisting one's sexual urges, it's about not having any to begin with.