Welcome to the forum TR! I'm very sorry to hear about all the difficulties you are struggling with in the relationship with your husband.
I'm not sure how else to come about to it, so I'll be blunt and to the point. There are a lot of red flags in your relationship, of which poly is only one. It sounds throughout your whole post as if your husband has been a spoiled child for the vast majority of your relationship together, and that you let him walk all over you and quietly clean up his messes behind him. You don't address your needs to him, and if you do he fights with your or flat out ignores them. He does not seem to care about the wants or needs of anyone besides himself, which is a very alarming quality in a husband and father.
If he is not willing to consider relationship counseling, I might suggest going to some counseling on your own without him. The relationship between you two seems to be flirting with the line of emotional abuse. Your husband always gets his way, and he is very good at avoiding or squashing your complaints when they arise, painting you into the corner of the "nagging wife" that is ruining all his fun.
But what happened to your fun? You are a human being that has needs, wants, and dreams just like he is. You BOTH have children that you have a responsibility for, as well as your responsibility to each other as husband and wife. It isn't reasonable or fair for him to expect you to do all the work and to make you feel guilty if he isn't being entertained or pampered every hour of every day.
It sounds like he may very well attempt to have a sexual or romantic relationship with this woman regardless of whether you approve of it or not. He will probably badger you until you give some kind of white flag to him that amounts to "Do whatever you want." It is up to you to decide if you are going to continue to stay in this kind of one-sided partnership where you are always left holding the icky bag of doing all the work so that he can play for his entire life.
I'm sorry to hear that things are going so terribly for you at the moment, though. If poly isn't something you want, it isn't something you should be forced to accept into your own life. It is hard to even sift through to the poly itself, though, as it seems like there are quite a few other really concerning problems with your relationship already. Trying a polyamorous lifestyle can be difficult even for very healthy and well-balanced couples. Trying out a polyamorous lifestyle for people who are already on the rocks usually doesn't end well at all.
I wish you the best of luck and hope that you are able to find some good advice here, even if my post doesn't help. I'm relatively new to this forum and there are some really great posters on here that seem to have a really good way of explaining concepts related to relationship dynamics and health.