I'm VERY sorry that you have to deal with this. I, unfortunately, have to agree with the other posters. I don't think this is a case of being poly as it is he wants to do what he wants and is going to use being poly as his excuse. Anyone that's in a successful relationship of any kind will tell you that TRUST is one of the most important foundations. From what I've read, you don't have that. Worse yet, he's not willing to do what he has to to regain that trust. Telling you that your sex life would have to suffer?? It doesn't have to and shouldn't.
You have every right to feel the way that you do and he shouldn't undermine those feelings. Being in a poly relationship is extremely tough work and requires EVERYONE involved to do their part. I'm sorry you have to go through this, but remember that every life experience teaches us something. *hug*
Live life to the fullest 'cause you never know if you're gonna wake up tomorrow!