Ok, I will be blunt, sorry in advance. But honestly: Stand up for yourself. If you give in now, as you seem to have done before again and again, nothing will change. You are absolutely right to feel hurt and unimportant to him. From what you describe here, that's exactly the way he sees you. Given the point that people can be too shocked to see the truth or the appropriate way to handle situations (just experienced something similar) if you mean anything to him he will come around and at least talk to you. If this matter is more important than your relationship and he stubbornly insists on his selfish demands, I wouldn't keep up with such a selfish person.
His ultimate threat (divorce) seems like a prank. What does he gain by divorcing you? If he loves you he should be hurt by this as well. This doesn't seems to be the case. He doesn't say: "I can't handle you being with another at all means. This isn't something I can do, my love works only exclusively and needs exclusivity." He says: "Sorry, but as long as I can't sleep with the one you sleep with (or maybe even who sleeps with you as he seems to be really posessive of you) and as long as you aren't beneficial for me in regard to get new sexual partners lining up for my pleasure, this is no longer of any meaning to me." I would ask myself: "Is this all our relationship means to him?"
I would advice you to sit this out. He won't expect you to do this. Normally you always gave in, show him that there is more to your wishes and your self respect than he expects.
So much for the aggressive part. If he really has other problems here except his ego and twisted vision of your relationship and your duties, meaning if he has a problem with jealousy or is generally overwhelmed by the situation, offer some middle ground. Keep it low with your girlfriend for a while. Offer to talk to him about his problems and to help him work through them. Don't do the same he is about/has already done to you by just running your head against the wall. You shouldn't give in and tell him to end all of this and that everything was a big mistake if you feel that you need this to stay true to your self. But offer him some time to adjust to the new situation.
It sounds as if there may be some bits of co-dependency or at least a wished kind of forced co-dependency on his part (wanting to be involved in your other relationships/everything you do by all means). If you get the chance to talk to him try to get the message through that you are two persons, not a wondrously mingled couple-person. Each of you needs different things. One can't shape the partner the way one wants. This becomes unhealthy pretty fast if you try to do so excessively.
Hoping this makes sense and I got the picture rightly.
Facts: 30, female, bi, v-type relationship with Sward (husband, straight, mono) and Lin (boyfriend, straight, mono), poly-fi and co-primary.