From what I can see you are getting some good advice from many of the members here . . .
I have been married for nearly 8 years in a mono-relationship, and my husband and I are just now exploring polyamory/polyfidelity. I can tell you that making the leap from a mono-relationship to a poly-dynamic takes more than just the decision of one person. Sure the person desiring poly needs to have their feelings respected, but not if it creates a toxic situation for the person they are already with. A leap from one-to-many, or even from one-to-two, needs to be discussed and agreed upon between the two of you and not something that you should "just accept" because this would be a selfish act on his part and a disrespect for the love you have together.
That having been said, with all the problems you have been experiencing it seems that he is responding to the issues in your relationship like a child with a broken toy: "its broken so I want a new one". You cannot have a healthy new relationship and maintain the old if the old one is broken. If he doesn't want to address the hard issues in your marriage when he has you, kid/s and one on the way . . . what says that he will be able to handle the rough spots that might form in a poly-dynamic?!
I know that you mentioned that he doesn't want to cheat but chosing a poly-lifestyle to try and circumvent "cheating" is not a good idea under ANY circumstances. Poly takes more than just the idea to have another person in your love-life - it is a mindset and intent that takes effort, honesty (with others and yourself) and exploration into the limits of each person's capacity to love (which is different for everyone). If he goes ahead with his desire for another woman when you are not on board or approving, HE IS cheating because you desire monogamy from him. Just cause you know about it will not make it any less hurtful . . .
Also, he may not want divorce because of the kids but not wanting to fix the problems in your marriage and just allowing them to fester WILL HURT the kids eventually anyway. By staying together in a "go nowhere" hurtful situation the adults will be making a toxic situation for the kids to grow up in. If you have sons they will see that it doesn't matter what a woman says because a guy can do what he wants, and if you have daughters you teach them through example that it is okay to be hurt by a man and that fidelity doesn't exist . . . your kids learn from the situation that marriage doesn't work and their own relationships later on will reflect the fighting and yelling that you do with your husband.
If he won't budge to heal your marriage and you do not want another woman in your marriage, then I can only see divorce for you. I know it sucks. I know it is not the most ideal choice but if he will not help you, then it is all you have. You cannot save your relationship alone - it takes everyone involved . . . whether that means two people or seven.
Much love and peace of mind to you, my dear. I hope you are able to sort things out and get into a healthier situation - for you, your kids and your husband. <3
"Love is the condition in which the happiness of another person is essential to your own." ~Robert Heinlein