All I see on his part is selfishness & manipulation. Granted, all I have to go on is what you say which is, so very understandably, colored by your emotion.
He is determined to have his cake & eat it too, regardless of your feelings. I'm angry for you. Not AT you, as he has chosen, but for you; that you are being apparently selfless for this man who shows none of that consideration for you. I could not do what you are doing. I could not forgive the lack of remorse from him & her. Yes, I believe poly can work FOR SOME, *if* - and I feel this is crucial - the playing field is level & all are in agreement. I would flip every time he left to see or spoke to this woman.
I am absolutely dumbfounded by the situation & I am so sorry for you. This is not something you should be feeling, if it's the right thing for you. Obviously there will be tough times no matter what life path you choose, but overall? Yeah, this ain't right.
** ETA: I would (probably) ask permission for the same freedom. How would he really feel if you were seeing another man/men??
Steph: Wife, Bisexual, 24 year old mom to 1
Patrick: Husband, Straight, 23 year old dad to 1
Currently exploring Poly together, our latest adventure.