There is some great advice here in this thread, it was helpful to read :-)
I too am a pretty non-confrontational kind of person, and had the unpleasant experience of getting chewed out by the best friend of one of my partners. He told me that I should stop "doing this to him", and that I needed to decide whether I wanted to be with him or not.
I told my partner about it later, and he was very supportive and spoke to the friend in question about it. From here on in I know I can be polite to this person if I have to see them socially, but will no longer engage with them on any level above civil pleasantries. At first I thought that this was best, because it made things easier for my partner, but then I realised that I didn't owe this person any kind of explanation about how I lived my life. I can understand that he's concerned and wants to look out for his friend, but at the end of the day, he's not in this relationship, and it really is none of his business. It was really freeing - to decide not to expend any more mental energy on what someone else thinks, and just get on with my own life.