I'd say the two of you are simply incompatible. That he would expect you to do all of the changing to match his preferences--while calling you selfish for wanting to negotiate something that allows you to seek fulfillment based on your preferences--is a warning sign. That the only way he has supposedly contemplated a poly thing working is by placing severe restrictions on how it would manifest for both you *and anybody else who would get involved* shows he's exceptionally selfish.
He has every right to decide what he does. He has no right to decide what anybody else does. From what you've described, he's indicated that the only way to be involved with him is if everybody does what he decides.
That's reason to walk on and not look back.
I don't think every relationship should be saved, nor do I think that every relationship is worth exceptional amounts of work to keep going. I've been around long enough and involved with enough people to have experienced the epiphany that there are always more relationships possible and that the work involved with some is just not a good investment of my time and attention.
In this case, if he insists that his mono inclinations require that you be mono or that anybody you get involved with has to essentially be mono with him, too...well, I don't see much of a future there. Yeah, you could stick it out and couple years down the road he might realize it's not all about him and loosen up a bit, though that's a crap shoot. Or you could just find somebody else who's more compatible and avoid the unnecessary conflict over that time.
Compatibility is not measured by love. All the love in the world won't make for a solid relationship if the folks involved are otherwise incompatible. I have friends whom I love dearly and with whom I know I couldn't have a romantic relationship (despite all the sparks) because I know we're incompatible.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.