My views are obviously tainted by my own experience of dealing with polyamory while not having sex with my spouse. And it SUCKS. Big time. We had a good sex life for 13 years or so, so our situation is very different than yours, but I still wrestle with these ideas of being displaced, demoted, etc. And in my case I'm about to toss W to the curb, not just for the no sex, but for many other reasons. If you have a few hours to spare, read my thread for a not-so-pleasant read.
I can feel from your initial post your fear. And who knows, with enough understanding, study, introspection, etc. you might learn to be OK with a setup wherein your wife has sex with others but not with you. But consider this: If you feel inadequate and threatened before you guys even open your marriage, how do you think you're going to feel the first night you're lying in bed alone while thinking about your wife having sex with another guy?