Perhaps some kind of geometric pattern that shows your actual configuration, circles/squares connected with lines. Obviously you wouldn't be able to display actual male or female symbols without giving it away. I guess that would have to be a brooch or something, don't know what the guys would wear.
It's definitely possible to do this kind of stealth thing. The Invisible Pink Unicorn
is a stealth symbol of atheism, and the logo design is very stylized, looking only vaguely like an actual unicorn. I've worn an IPU t-shirt around my fundamentalist mom and she didn't have a clue. She didn't even ask, but if she had I would have told her it was from a video game or something. Or a logo for a company I did contract work for.