Its the Hi-fidelity effect. Often we assume our partners are off having "the best sex evar!" with others. Cruel imaginations. I think we all face this at least at first.
Something my partner and I have discovered is that newness is often hotter in theory or initially. The whole thing about reality rarely living up to the fantasy. It can be fun, hot, enjoyable for sure. It also includes having a partner that is still learning your timing, your special spots, might not be into everything your longer standing partner does that you enjoy, and wants things you might not enjoy that you don't have to deal with in your marriage.
The dynamic and personality differences often highlight exactly why you have your spouse in your life as deeply as they are and not someone else. Notice she has to WORRY about SCARING them off. You give her acceptance and peace that they cannot. You are united with her on many things they maybe cannot or will not care about near as much. What we've discovered is after a bit of time with others, it can be a relief to go home to someone you identify with better. I use to feel guilty about that and wonder if it meant I was "hard wired" mono until I heard my husband expressing the same. Despite all the reasons why your secondary, as nice and fun as they are for what they give, can also get on your damn nerves after a while. And while your spouse can get on your nerves too, you likely wouldn't have chosen to share a life together all this time if they got on them too much.
You can spend all your time dreaming up the worst case scenarios and yes they are possibilities. You've had all this good in your years together - yet you torture yourself thinking the bad outcomes are more likely to happen between the two of you and not one of you and someone else? Its more likely that you know more about making your wife happy and what she values than someone she just began to connect with.