You reap what you sow.
You had a committment to your original partner. You broke that trust/deal. Love or no love, right or wrong, regardless of intentional out-come. Humans make mistakes, and hopefully, we learn from our wrongs. Cheaters are not neccessarily jerks or bad people. Many times, they are people who made genuine mistakes.
However, you broke trust. You probably lied, and you definitely hurt your partner worse then you can imagine.
If what he is saying, is that he can learn to grow into accepting you as poly, but cannot handle you doing that with the person you cheated with....why not right your wrongs ?
I have rarely met a cheater who wants to actually be accountable for their actions. They want to cheat, be forgiven, and carry on with both partners. They always profess they can`t live without the love, of the partner they cheated with. Where, rarely are they asked to give up loving that partner. They are asked to QUIT the relationship, nsa, and work on the marriage problems that led to lack of communication and cheating. No relationship, is definitely not a forever statement, when someone is still hurting with pain.
It is always full of drama, when someone is determined to keep both partners, and I have yet to see a good out-come. There is always this up and down yo-yo effect, and nobody is truly happy.
However, on the flip side, I know of someone who cheated, professed to love both the original partner and the mistress deeply. Wife asked him to give up the mistress and work on their marriage. Wife was initially adament, that she never wanted 'The other woman' near them again.
So he did. Threw himself into fixing the wrongs.
A year later his wife told him he could invite the mistress back into his life, as a girlfriend instead. They took their time, went slow and did things 'right'.
They are still a unit, 15 yrs later.
So I would say, it is AMAZING the good things that can happen, when you DO RIGHT BY PEOPLE, even when it is hard to do. Even when it hurts. Even when you suffer.
If you had a situation where you 'righted the wrongs', and the spouse still didn`t give you a inch, then you might have a leg to stand on.
Good luck to you, either way.