A metamour who sees you as a friend is invaluable. It might be too much to ask of the world/fate to find that, but that doesn't mean that one should settle for the opposite extreme, a metamour who sees you as a threat/enemy. How stressful.
And it kind of irks me that he wouldn't be willing to stand up for his other relationships and say "No, it's not ok to keep changing the rules, that's not fair to anyone." Or, he could always say "Y'know what, if this is how things are gonna be maybe this marriage can't handle poly at the present time." But instead, to just let this bad situation continue... bleh.
I think not being with him is the right decision. If that means you need to be the sort of friends who don't actually see each other in person right now... so be it. You deserve better. Take whatever space you need.