If everyone is communicating and there is talk of what this man means to you with your husband then there is no reason that this is "wrong." There isn't really "wrong" in poly relationships, the joy of them is that you can arrange them how they suit you and your loves. It's about creating what works for all of you. Not just you or one person in particular. That is why there is so much honest, open communication from a place of respect of each other and why there is so much boundary negotiation.
I would wonder what is going on with you husband that you are not intimate anymore. That is a bit of a red flag for me. Just because you are not intimate with him does not mean that he has no attachment to that notion that someday he will be again or sex with you is lacking specialness. He could be very hurt if you are seeing this man behind his back, not only the sex part of it but the connection part of it. That holds just as much value most of the time.
Maybe its time to fish or cut bait with this situation... either leave and be with this other man or start a life as an independent woman, or try out a poly life relationship and get talking with your husband about what kind of arrangement that leaves you both in. Lots of work to do regardless... sounds like a big change is about to come and that its about time...
Hard to say with such limited info. But hope that helps.
Anyone want to be friends on Facebook?
Send me your name via PM