You should talk to your husband for sure. Its odd you would hide aspects of who you are from the person you're building a life with and then share those hidden things with others. Do you see how that sharing of secrets might create an atmosphere of attraction the way people who end up in affairs often do? Then you have to wonder how much of the growing attraction is due to the secrecy you don't have to maintain with Pan and how much of it is based on real compatibility.
Nip it in the bud by eliminating the secrecy.