For me, what I would find most upsetting is not the trash talking by others but the lack of defense by your primary. Obviously give trash talkers the boot. Maybe a warning first if so inclined.
I also do not enjoy confrontation. Makes me nervous and sweaty. But I do it when necessary because well, being a grown up. Your primary partner needs to step up and shut down that kind of talk about you. It doesn't have to be loud and aggressive. Usually a quiet 'That comment is unacceptable. Stop.' is enough to halt almost everyone in their trash talking tracks.
Also I am a little amazed this seems to happen kinda frequently to folks. When I was with my ex, no one trash talked her in my presence ever. Is this common?