What strikes me is that you are very focused on what people think about you and that is getting in your way, as far as being able to communicate directly with others. You've said here that you don't want to come off as bitchy, demanding, a pansy, confrontational, unusual, looking like a groupie, rocking the boat, and so on. All that self-conscious monitoring of how you appear has come back to bite you in the ass, my dear!
I don't understand why, after a year of seeing him and socializing with his other gf and her husband, you still don't feel able to talk to both of them together. Look, it's just talking. The world won't come crashing down if you express yourself.
You have been sitting on being uncomfortable for the sake of looking like you are "cooler" than you are, and now you have to back track and speak up.
Hey, it's okay to be uncool. It's okay to feel whatever you feel, and you don't have to set your feelings aside for appearance's sake. It's perfectly fine to say to both your bf and his gf, "You know poly is still new to me, and I have tried to be as comfortable as I can with different aspects of it, but I am realizing now that I would feel better with a boundary around public displays of affection between the two of you when we are all out together. It makes me very uncomfortable when I am with my boyfriend at a party and he is physically affectionate with someone else, because I'm his date for the night and for me that is special time to be with him. I am working on feeling better about it, but would you both give me some space and keep your affections private for a while until I can handle it being out in the open?"
Saying something like that clearly isn't a ban altogether, but says that you are willing to see what's behind your discomfort and are simply asking for some consideration while you figure it out. And then follow through and do the self-work to figure out why it bothers you so much.