Hi, thought I would chime in as an employment counselor for the developmentally disabled. I am sorry that you felt marginalized by Boring Guys's comment. I personally did not find it ableist, but I respect your right to feel what you feel. I think the problem here is that instead of owning your feelings and using non violent communication, you called him ableist. That was not particulalry endearing or constructive. Try learning about non violent communication. It may help with your husband as well.
As for the sex parties, you can have your opinions, but as far as not having them in your life, I don't know. Maybe for that one aspect you should have a DAT policy? Because as long as they are dating, it will be in your life, even peripherally and could come up in conversation. You cannot ask a metamour to stop participating in something you don't like just because your perception of it is negative. Your husband can choose not to participate but that does not change much since I am sure they will still talk about it. Also, he may grow to resent the agreement to box up his more adventurous sexual nature to cater to your sensibilities. I know Runic Wolf resented losing the bdsm aspect of his first poly relationship when I discovered his lover was a cowgirl and flipped shit. He and I have some different kink needs that overlap in some areas, but the situation tainted my perspective and desire to explore those areas for years.