You can be penetrated by a woman. Whether you're male or female.
But to answer your question, yes I can be more on guard when males are around than when females are around. However, since I'm a straight female, they are also the category of potential mates. Females I see as de-sexualised (as far as I'm concerned, since there is no attraction) and not a physical threat, so I guess I don't pay much attention to them.
I'm talking about strangers, here, which I believe you were too.
There is definitely some level of patriarchy at play there, although on the other hand it's true that if I get assaulted, it's more likely to be by a man than a woman. Also, if a woman assaults me, I'm more confident in my ability to fight back because I'm not particularly strong vs the average male, but I would probably be fine vs the average female.
As a male though, you probably have a lower risk of getting assaulted by males, so if you're constantly tense and on guard around them, I can see how it might be a problem. Especially if it remains the case after you actually know the people.