I don't know what else to add other than I'm sorry you continue to struggle. I don't think either of you is horrible people, but I don't understand why you both continue to try to make something go that just...won't.
His feeling annoyed that he has to watch children to free you up to see your BF -- well, that's reality. There are children. They need watching. Because you can only see BF on overnights once a month. Could DH organize a babysitter so he has that time off too? Or could you organize a babysitter? That's a side track thing to me.
Because I just think even with a separate babysitter he still would not feel good because he's just not poly or poly-friendly. He's mono. That doesn't make him a horrible person. Being mono is totally fine. Buy why does he stay in something that is not feeding him the way he needs to be fed?
Why do you?
Why continue in this manner? You both are suffering. It's so sad and tragic.