That does sound tough.
Am I right in thinking that your husband has been having an affair while you are 7 months pregnant and have 3 other children to look after? And now he wants to carry on seeing the other woman and you and he both want to be okay with that?
Four children is a lot to look after. How much does your husband help you?
I have a family member with 4 young children and she and her partner do manage to have a life outside of the family. They aren't poly but they do have friends and interests outside of each other. They work hard at it. Keep a big calendar and are both active parents. They both have weekends and holidays without each other while the other does all the parenting.
Is this something that your husband is willing to do?