My take on this is that it's his family, not yours. You don't have the right to decide whether or not he is out with his family, and whether or not he chooses to visit his family.
They may be ass holes, but they're his ass holes. It's not loving and supportive to tell your husband that his family is a bunch of ass holes whom he shouldn't be allowed to visit because doing so makes you uncomfortable.
Now, none of that means you have to go with him. As much as you don't have the right to prevent him, he doesn't have the right to force you to join. There's absolutely nothing wrong with married folks going on trips without their spouses. My husband and his daughter went on a Caribbean cruise last Christmas. I dislike heat and humidity so I didn't join them. I went to my home town and visited with my family instead. It worked out wonderfully, plus when we rejoined, we had stories to share.
“As I am sure any cat owner will be able to tell you,
someone else putting you in a box is entirely different
from getting into a box yourself.” —bisexualbaker