I have never said (or written) that everyone can or should be in a polyamorous relationship. I say in my book that a good idea is to take inventory and work out what is really important to you. From that inventory you can work out what you really want in your life.
From what you say you want to be loved passionately and you want the material and financial benefits that you've built up over the course of your marriage. I don't think those things mean that your marriage to a polyamorous partner can't work but I think your abandonment issues will make it more difficult.
Your relationship with your wife seems problematic if you feel that she is giving affection to another party and not to you. I think that polys in relationships with monos need to work very hard to help their partners feel loved and secure. Do you think that she is doing this or do you think she would do this if you allowed her to? If a person is very angry and hurt it is sometimes hard to love them.