This is something I've been struggling with for a long time. With my home life I never got out much, and my father put a huge early dent in my self-esteem. I always think I'm not smart enough, strong enough, fast enough, attractive enough, to fat, or in other words, always lacking. Because I have this view of myself, I'm extremely self critical of any perceived failures, personal, social, or with relationships.
How do I fight it? It's hard some days. Sometimes a massive "failure" can send me tumbling deeper into a depression cycle for my bi-polar. But there is one thing I always do that helps me gather my self-esteem up.
I get up, dust off, and have another go.
The worst thing you can do is fall on your ass and not get back up again. Trying something over and over again can hurt, but the feeling of beating it can help build more self-esteem then all the failures combined. As was said before, learn a new skill, or try something new and see if you have a talent for it.
But most of all, believe that you can do something and keep on trying. Don't know it, feel it.