This is a tough one. I don't know enough about poly yet so maybe I shouldn't even be answering.... but it sounds like your marriage is pretty strong in all other areas.
I think what might happen is that intially you will be quite happy that his sexual needs are being met; but that when you will feel threatened is if you see any signs that he is enjoying an emotional relationship with the new partner.
On the other hand, this big thing (you worrying about your sex drive) has been hanging over you for a long time. You might even find that taking the pressure off for a time, your enjoyment of making love might start to come back.