I'm going to go out on a limb and say you are being affected by the all-to-common disease of competitiveness.
When you thought you were # 1 in the sexual fulfillment dept - you were golden.
Now you see some potential to have that position challenged - whether it's real or not.
Repeat to self 1000 times ............
Love and sex are NOT a competition !!!
When you see his mate as a partner (as opposed to a competitor) working towards the same goal (his happiness) these feelings and fears will go away.