the only way I can see this working for me is if we had someone we were dating together, so I didn't feel completely left out/left behind/unwanted/thrown aside/ignored.
Do you personally know any people in successful poly relationships? Maybe you would benefit from seeing some examples of it working. And talking face-to-face about your fears might be more productive.
Have you looked for poly meetups in your area?