To some people it does seem to matter - I have never been able to figure out why. And if you think even briefly about 'the number' it quickly becomes clear it doesn't really tell someone anything useful to know about someone's sexuality. It doesn't indicate if they like sex, why they have sex, what their limits are, what they like.
That said I am curious about people's numbers. I often find it is a conversation starter that can lead to the more interesting stuff. Of course, I don't start with this question if I don't know someone and even if I know them well it is usually easy to figure out if sex is something they feel comfortable discussing.
Interesting question OP! I wonder if there is a poly version of this? How many partners (however one defines partner) is too many? And obviously the answer is very individual.