Sometimes I don't like to bother with any of the consequences of short-hand names for certain ways I chose to live my life. I call myself poly, a kinkster, and an athiest, in most cases. Sometimes I don't, and I don't think it's a result of insecurity, identity-crisis, or a desire to not be boxed in. At least not as some others have indicated people use "boxed in" as an excuse.
There are three reasons I ever feel any discomfort about applying one label or another to myself.
First, I have no control over what other people using the same label do.
Second, once there are two people with the same label, there follows this concept of community, how we share this label together, how we should automatically have some sort of connection, common purpose, and reason to protect the "turf" of this label community.
Third, I may not be interested in communicating about issues surrounding the label if there are more important things to discuss and the label might derail those points, or poison the well for them. If the shortcut of language that is the label would not be understood correctly by the audience, then the label hurts understanding more than it helps it and you might as well abandon it. (Or decide get on your soapbox and try to educate.) If everyone around you thinks "atheist" means "devil-worshiper", or "poly" means "LDS cultist" you can decide to use the label and possibly derail your point, or you can avoid the label and try to focus on the real topic.
Whilst we are all entitled to self-identify as we see fit, we are not entitled to pass judgment on other people's entitlement to do the same.
And I agree with this to a point, certainly everyone has the right to call themselves whatever they want outside of legal issues like calling yourself a doctor, lawyer, accountant, Realtor(tm), or another licensed professional title without a license. But there comes a time when someone is ignorant or deliberately lying about this or that label they have given themselves, and then it becomes necessary to judge. Possibly, you should judge yourself first and ask whether the label truly applies to you. If it does, and the person's behavior is diametrically opposed to your behavior, how can you not judge? To not make a judgment at that point would be to render that label useless in your identification. If person A claims to be vegan, but eats jello, and person B claims to be vegan and does not eat anything containing gelatin as it is derived from animals, then it seems obvious they have a mutually incompatible understanding of the label they've individually chosen for themselves. Maybe neither of them is right or wrong in the grand scheme of a living language, but for at least one of them, the label has become useless as a shortcut for explaining some of the ways they live their life.