I also don't think it's a bad thing to fantasize in general (spoken or unspoken), but it just isn't for me unless it's someone that would be okay with the objectification (i.e. someone he (or we) meets specifically for casual sex vs. someone he meets to have a relationship with). The person met for casual sex expects to be thought about in purely sexual terms so I don't feel bad at all for fantasizing about that person in various situations, regardless of whether or not they would ever happen. The person hubby meets for a relationship, though, is expecting to be treated as a partner or at least someone closely involved in his life. I wouldn't use that person as a sexual aid even in my mind unless she approved it beforehand. It would feel like objectifying her as a sexual object instead of showing her the respect she deserves. Now, if she got flirty with me or showed any other sign that she would be into me, I would find it completely appropriate to fantasize about her and hubby with or without me being involved since she instigated a sexual aspect to my relationship with her.
I know not everyone feels this way, and I don't expect them to. It's my personal take. I find it odd that I feel this way, honestly, since I'm not usually one to talk about objectifying people in anyway. lol