I really don't think that this sort of problem is unique to polyamory - I think that it can happen in monogamous relationships a lot too.
Your statement that you could get sex from any of your other partners, but it's not a replacement is, for me, an excellent example of how the poly mind works - one partner isn't just a plug-in substitute for another.
Does Lady admit that there is a problem, here? If so, then I would recommend spending some time with a couples councilor to have the room and the time for each of you to talk things through. There are many, many causes for a sex drive to fluctuate wildly - he may know what it is, and not feel like he can share for some reason, or he may have no idea, and a professional may be able to help.
Please check out The Birdcage - an open, friendly Polyamory forum for all parts of New York State
"Listen, or your tongue will make you deaf." - Native American Proverb