I think you are right AC. You are fast approaching a point where you will likely just have to decide between poly and your husband. You can't change him and make him do something he doesn't want to do. You can't make him do the work, but you can decide what you will do and see what happens.
I suggest you get some things in order first; place to stay, some money, some written agreements if you can on details about the kids maybe... If it hits the fan and he kicks you out you will need these things to be in order... then see what happens if you tell him you are moving forward with your men and that you will be discreet but it has to be this way for you because you are miserable and can't live the way you are any more.
Anyone want to be friends on Facebook?
Send me your name via PM