Hi mostlyclueless. If your agreement is to date people together then I don't think you're being unreasonable to voice your opinion if you think someone wouldn't be compatible with you or a good fit for the relationship.
But you're right that you may never be poly enough for him. As it stands now it seems that's the case, so unless one of you changes or you both change a little to a middle ground, it will remain that way.
Have you tried communicating through writing and then reading what you wrote out loud to each other? I find that for sensitive conversations with the potential for argument, this is a good way to go. Writing allows you to choose your words carefully, express yourself fully. It ensures that the other person listens to everything you have to say before responding (and that you do the same for them). And by reading what you wrote out loud to each other, instead of sending it in email or IM or text, you get the benefit of full communication (body language, face expression, tone of voice). If there's any solution here it will require excellent communication.