I'm not a virgin, but I'm asexual. If all my partners were okay with not having sex with me, I could totally have several non-sexual relationships. And that would be the best for me!
So, it's possible to do polyamory without having sex. But just like non-sexual mono relationships are rare, non-sexual poly relationships are also rare. Just like mono people may be turned off by virgins, it can happen with poly people too. But if staying a virgin is what you want, then by all means stay a virgin. You have the right to have the exact kind of relationships you want!