"One major part of polyamory seems to be the sex"
The only reason for this is that one major part of romantic relationships is generally sex. Of course, it doesn't have to be. There's nothing special/different about poly in that regard. My girlfriend and I went more than a year without having sex, while she was married and I was seeing other people, due to issues surrounding her libido in relation to her pregnancy.
Mags has a point, though. If you're in love, and very attracted to someone, and you both are being physically affectionate, touching, kissing, almost certainly turning each other on... do you really think it'll be enough, for you and the other person, to go home and masturbate while thinking about each other, time after time, while never taking the opportunity in front of you in real life even though you both desire it... for *years*?
I can't tell you what's right for you. All I can say is that, for most sexual people, physical intimacy makes them feel loved and bonded, and forming a long-term loving bond in which that form of affection and fulfillment is denied to you for years on the sole basis of the fact that you haven't earned it yet... well, I think you'll have trouble maintaining functional romantic relationships with sexual people, and that'd be true whether you're mono or poly. Fairy tales are for children for a reason -- real life doesn't work the way it does in books. You can still have loads of romance in your life without letting romantic notions dictate your choices.
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.