When Peaches first mentioned poly to me, I think I felt something similar.
For me, though, the fear went away when I fell for someone else. I realised that me loving someone else didn't change my love for Peaches not one bit. And that I couldn't really tell who I loved best. That each love was unique and couldn't compare.
So I figured if Peaches ever fall in love with another person, it will be the same: He'll forever love me just as much as he loves me now, and I'll always be special to him in a unique way no one else can. Just as he and CC are to me.
That's all I can say with my (little) experience in poly. I hope it helps, tought.
And I don't think any fear is silly. You should pay atention and vallue what you feel, and take things slow if necessary. Talk to him about, too. I'm sure he'll undestand.
Me, female in a V with Peaches, live in boyfriend and CC, boyfriend.