I feel for you so much. I really feel like from what you are telling me that perm's poly relationships were in reaction to your poly relationship with cue. Of course I could be way off, but from what you're telling me that's kind of how it looks, since two of the other women in his life were quite attached to him and that distressed him. Sounds like his ideal would be you two having a pretty close marriage with a few short, somewhat superficial relationships on the side. This doesn't seem to be what you want. Regardless of how cue may not be a perfect person (who is?), I have the feeling that if you were in a relationship with somebody as close to perfect as you could get, perm would have problems with him.
I hate ultimatums, they force people to make decisions they don't want to make. You're in a really tough spot here, and my heart goes out to you. I'm not going to try and take sides, but you have to be true to yourself. Are you going to be happy if you have to live the way he wants you to live as a couple? Is life without cue going to change your marriage in regards to intimacy and one on one time, or is it wishful thinking? Will you resent having to break it off with cue? How will that change the relationship? It sounds to me like you are trying to compromise, but it's not enough for him. You have to figure out where your line is and stick to it. Just as he has to figure out what his line is and what he can live with. I know how you feel about the marriage fixing thing, I wouldn't call myself a Christian but I seem to have inherited the guilt and desire to sacrifice and fix things from my family, and I know how hard it can be to ask for what you want. Personally I think you need to figure out what is right for you and what makes you feel happy and loved and go with that. I know it's not that simple, but if I have any advice to give, it's to be true to yourself.