You've described the classic spousal abuser in your post. Not physically abusive, but by making you feel completely dependent on him and getting to agree to no outside social contact (which is unnatural for anyone, poly or mono) you've allowed him to gain emotional control of the relationship. This doesn't make you stupid or gullible; manipulative people have a natural edge because they're willing to intentionally manipulate others, whereas most of us have accepted the idea that manipulating others is wrong.
Contact your family. You need some emotional support, and families are willing to provide that. You need a lifeline, someone other than the internet to pour out your problems and worries to, and families will listen. You need ideas on what to do, and if your family is like most, they'll have ideas. (Mind you, you don't have to follow those ideas; but sometimes, bad ideas will spark good ones. Just some suggestions on what to do next, however impractical, will do you a lot of good.)
Most of all; you're not alone. You're not the first person this has happened to, nor (sadly) will you be the last. Dedicating yourself to someone who has proven unworthy of that energy may have been unwise; but it doesn't make you a bad person. You, and your children, deserve a better life. Make sure you (and they) get it.