Give me a traditionally HAWT man with a crummy personality, and I'll show you someone that I have no interest in. It does go to the point of affecting sexual desire. Perhaps this is why the vast majority of casual sex I've had has always been with friends? I find the act of trying to pick up at a bar unattractive, and so everyone in the bar just goes down a notch for me. (Even if I might have also been trying to pick up, not necessarily get laid.) Funny how the brain works!
Now women on the other hand ... I will immediately find a woman attractive! In terms of the strangers I check out, women make up 95%. Women tend to work backwards, if I find them immediately attractive. (If I don't find them immediately attractive, they follow the same path as a man.)
I think I go backwards on this. Men, I absolutely check out, and I can appreciate the aesthetics of a fine man pretty easily. And again, if he opens his mouth and pearls of nutcaseness fall out... I find myself very turned off and wish he wouldn't have said anything.
Women I think I have to know really well, and be compatible with, and I'm still very slow to warm up to an "attraction" level. Slow enough that I find it simpler to call myself gay rather than bi.