I think poly people behave many different ways in social situations. Some are very comfortable hanging out with their lover's lovers. Some don't feel that way and choose to do something else. I think you should do that which makes you the most comfortable.
I sometimes start to second guess myself at parties. When that happens, I tend to overly analyze everything. When I see myself start to do that, i try to break out of it by going and doing something I haven't done before. I figure if I am in that state of mind, I should at least do something original so I can think about it later.