Since you felt sad that your wife was not able to work out a relationship with another is a good sign that you have compersion already. So maybe you just have to try to understand your other feelings to let the compersion flow through more easily. I found that I can feel more compersion the more I try to put myself in my wife's position. Compersion seems to me to have empathy at its core.
So maybe if you focus on dealing with jealousy, you may be able to handle the Dr. Jeckle & Mr. Hyde feelings better. For that you can try to find out if you are feeling insecure or envious. If you are insecure, what specifically is it you are worried about? Once you can have a strong grasp on what bothers you (like maybe you are worried that her other lovers will be better than you at something), then you can talk to her about it. She can reinforce what she likes about you.
Over time, you may be able to see better that she is with you because she wants to be with you. She is not fulfilling some old vow or just hanging in there. If she is with you, it is because she values something in you that keeps her there. She has no reason to leave you unless you are making each other miserable.
I think you have to be careful with some statistics on open marriages. For example, regular vanilla marriage has about a 50% chance of success. That is Russian roulette with a half loaded gun. I think polyamory has a big advantage over many other relationship models. It heavily pushes communication more than in regular monogamous relationships. (I have heard several stories of monogamous couples buying polyamous litature because it does a great job of helping people open up their lines of communication.)