It could be that you aren't cut out for a poly relationship or it could be that this particular woman didn't jive with you. Take some time to work out the reasons behind what brought up the emotions for you while he was in the secondary relationship.
He's been very clear with you right from the start that this is who he is. That isn't going to change. If he is poly there are going to be other people who he is going to develop feeling for and relationships with. It's up to you to decide if this is something that you feel like you can live with.
Everything will be ok in the end. If it's not ok it's not the end.