Hell, I'm not a guy, but I've run interference for people many times. I think the main point here is that being confronted with pushy guys can be very overwhelming for a woman (just speaking from my gender experience here).
In this case, it sounds like you didn't back your girls up. They all crowded closer to you, were trying to shut him out, even redirecting the conversation your way, and it sounds like you just looked at the guy and let him go on. I find it hard to believe that they were really okay with it at the time, even though they may have dismissed it later on.
I would stand up for a friend at the point they became uncomfortable, let alone a significant other.
Honestly, if it was me there as one of your women I would have told the guy politely to piss off, and then proceed to body block him from the rest of the group. If he was persistent then I would quickly demand his compliance--but I'm just aggressive like that.
The same approach would NOT go over well if I were the man. It would indeed come to blows, quickly and often.
It doesn't sound like any of the women in your circle is happy acting as an alpha bitch in situations like this. Assuming that I was uncomfortable and feeling more passive, I would have looked to you to step up and defend us as a group. I wouldn't want you to get in his face or anything, but I would feel lost if you stayed in a passive role.
This sounds like a scenario that's not likely to stop coming up, so I would discuss it with the women, decide what you all want to do. If they want you to step up and be firm or to let it slide. You should decide as a group some signal to say "that's it, we've had enough, and it needs to stop, even if we have to move away from the guy ourselves." Your uncomfortableness with the situation is an incredibly important topic, and the consequences of these confrontations (you possibly getting injured by an offended party) are much more serious than them simply being uncomfortable for a brief time until a guy gets the hint.