"He does get quite melancholy, depressed, jealous if he gets any impression that I just might be thinking about doing that. He says that he dreads the day that that happens and he loses the woman that he is in love with, me."
Considering that he loves two women, why would he assume that if you became involved with another man that it would mean he would be losing you? It makes me think that maybe he hasn't really accepted the idea of loving more than one person, and that it's only ok for him because of this special circumstance, his wife's lack of sexuality. And I can definitely see how that would make you scared.
It's not ok to be in a position where you feel like you are a very important person to him and a priority up until the second he gets her back and then who knows. You deserve better than that. I wonder if you can gently explain these things to him, maybe show him this thread, and explain that for your emotional health you have two choices at this point:
a) he tells her he loves you just as much as her and that it's not a competition and he wants to treat both of you equally, which means you stay a big part of his life even if/when she gets her libido back... I can see why he would be scared of pushing her away by saying this, but it's not fair to anyone to carry on ignoring the fact of your love and the possibility of her recovery, or
b) you continue to love him and be involved with him but actively seek out another love or loves so that you're not solely reliant in your emotional life on someone you can't count on.
Maybe that idea is too scary or maybe you don't really need it. But just remember, it's ok to be "difficult" sometimes if you're doing what's necessary for your heart. Better that you guys work this stuff out than that you suffer in silence to the point that you have to leave because it's too hard.
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.