I think he's using "attracted" in a different way from the way I (and maybe you, nycindie) use it.
To me, attracted means sexually attracted, in other words, that person or group of people can potentially make me horny, I can want to have sex with them. That's "attracted".
It seems the OP's partner is using "attracted" as in "romantically attracted" or something, as in meaning "wants a relationship with" or "wants to cuddle with".
I find it confusing too but I'm pretty sure that's what it is. I can't imagine someone saying "I'm not attracted to men but I want to have sex with them" and meaning "I don't want to have sex with men but I want to have sex with them".