My hyper-logical brain would explain it this way...
if you're an anonymous sex type person, then it is a compliment. Those encounters seem to be "hot or not" determinations.
if youre a "personality means something to me when it comes to sex" type person, then of course it's not a complement.
being either kind of person is fine and dandy, but your husband shouldnt expect you to be something youre not. Communicate with him and y'all should come to an understanding.