First off,..I think it`s a really good thing, that you A) recognize this as a problem you need to fix. B) don`t restrict the women in your lives, due to your problem.
As for the issue itself. I think you have a two part problem, that you already recognize.
First is the feeling of competition you get, with regards to other men. Do you tend to feel competitive with men in a work environment, or in sports, or some other recreational activity ? If anywhere in the rest of your life, your jobs, skills, or play, involves being 'pited' against other males, your brain could be having a very hard time, seeing men in any other way.
Getting really honest with yourself, will be the first key. You need to figure out if this is habitual thinking, ( What I said about the competition), or if it lies entirely in a insecurity, you can`t quite admit to yourself yet.
Secondly, you definetly need to adapt coping skills that are better suited. By instilling better coping skills, and MAKING yourself react better, you will already solve part of the problem. Anxieties only build, when we handle them badly. It then becomes a vicious circle, where the 'issue' is almost overshadowed by the feelings incurred during the reaction.
Dealing with the 'reason' will take some time and a lot of thought process. Meantime, make yourself feel somewhat better, by finding new coping strategies, that at least leave you a bit healthier, and more in control of your emotions.