Let's say A and B are in a relationship. A is great at cooking. B isn't. B is great at fixing things. A isn't. A good balance of power here is an agreement that says B doesn't tell A how to cook, and A doesn't tell B how to fix things. Looking at only the "A doesn't tell B how to fix things" part, it seems really unfair. A has no power. The same is true with B and cooking. The overall power within the relationship as a whole is balanced, though, because the 2 areas of power that are weighed heavily in opposite directions come together to balance each other.
My impression is you and your husband have an agreement to stay out of each others dating life. Sounds fair on the surface. Looking under the surface, it is discovered your personal boundaries are much healthier than his. That means your opinion of how well he is maintaining his personal boundaries with his girlfriend needs to be more important than his opinion of your boundaries with your two lovers.
People aren't equal. That one of the reasons we all love to come together. I would figure out FOR YOURSELF what kind of OVERALL relationship balance you want with your husband. Push him as far as the overall balance you want allows, to show/teach/pressure/anything else you want to call it, him to get his boundaries as healthy as they can be inside of that balance you decide you want with him.
Maybe you think you already have a good balance. You have a "deal" with him that says you can spend the time you want with your two lovers if you allow him to be... ahhem... stupid with his girlfriend. Does the good cook have to yield to the bad cook every once in a while when the family should be able to tolerate a bad meal? No. That's not the way it works. The person with the better skills is in charge of all decisions made in that area.
You have the better boundaries. Spend the time with your lovers that you want to and push him to have better boundaries with anyone he is with - within the balance you PRIVATELY (because he's not good with boundaries) decide you want with him.
sparklepop, your thoughts about what toxic people are capable of is spot on.