There is no number that is too high or too low really. That's a personal thing that no one can answer. Though, when my too many comes into play for me is when you have sex with people you don't care about. My body I feel is my temple and I want to share it with those I have/do love/d. I feel I would cheat myself otherwise.
Of course, I know some don't see it that way either. This is just my personal thought on the matter.