Your request is NOT AT ALL unreasonable. He has violated your trust previously and seems to have no real regard for any kind of boundaries you wish to set. If everyone can't agree on terms/rules, then it should be completely off the table. Anything otherwise shows a severe lack of respect, and that's unacceptable.
You can't attempt something like a poly relationship without a foundation of trust, and you don't have that with him. If he disrespected that trust before, he can and will do it again.
Best of luck to you, and I hope things work out to where you can be happy. Ultimately, that's really what any of this is about.