Little of substance to add after informed remarks by the others, except perhaps that your feelings seem understandable to me--or at least familiar, having some experience after 35 years of marriage with a number of the psychological depths a couple can plumb, qualified if not disqualified by the fact I've limited experience with extramarital relationships, am male and essentially mono (i.e., recently declared poly to my serially mono wife, who welcomed the news by announcing I was free to choose between poly and divorce).
That said, as GalaGirl mentioned it seems you may be facing the spectre of poly, which unlike casual sex with strangers is not fun and games, striking me rather as entirely a topic appropriate to Dali (the surrealist painter). By "poly" I mean juggling love for more than 1 partner through all of lifes' and loves' stages, and trusting one's partner the freedom to do the same (a little like finding the courage to give one's horse its head during one of those vertiginous descents into the Grand Canyon, if you like).
It's no surprise to me that your husband may have fallen in love with someone he's having sex with simply because for me (and hence probably a lot of men) love & sex are on the spectrum of the same disease (like a host of behaviours characterise autism, say
). One difference between his behaviour and that of a male in a mono marriage engaged in an extramarital affair is that it's with your full knowledge, stemming from his assumption of your tacit acceptance, if not your blessing (given that in such circumstances it's often better to seek forgiveness than permission). If you can still think of it as sport it may be no more than a matter of parrying the thrust, returning the ball.
While how your relationship eventually unfolds is a matter of how you and he deal with issues as they arise it seems you're on the right track; namely, you're striving first to be objective and second, to communicate. In my opinion (and somewhat in my experience) secondary love interests, no matter how intense, don't need to become a threat to the primary relationship and should not be perceived as a threat. If embraced in the right spirit they tend to illuminate the primary relationship in a number of ways, strengthening a strong bond if it already exists, in any event prey after a while to the same banalities as the primary relationship. Probably all of us sometimes dwell on worst case scenarios when time seems to teach ideally we should be lending support to our partner--being happy for him or her--but by the same token, in return deserving understanding of & respect for our sensitivities and receiving help to outgrow them. Easier said than done.
In my opinion it helps in the early years to conceive of things in terms of balance--even a sort of crass ledger, say, that makes sense to the logical half of the brain, that matches body count with points for style, depth & grace. As GalaGirl implies but if nothing else such a ledger is a board to pin our fears to in order to contemplate them.
Good luck, from the bottom of my heart. While polyamory seems to me the most free, natural & happiest form marriage can evolve into it's not for everyone (including me yet, as it turns out). If I might grasp your feelings I don't envy you them. From the vantage point of old(er) age sharing love in the early years seems complicated by what is to me natural (genetic?) competition for resources (emotional, sexual, material) and issues of dependence, dominance & security, a time when it's only easy to be benevolent when dealing from a position of relative advantage. The complications lessen after the potential for child rearing passes and career aspirations fade, leaving you and your partner(s) and their partners more clearly in each other's focus. If I miss youth & the newness of everything I don't miss the confusion & growing pains. The battle to forge a life together (if it turns into one) can be damaging but apparently true what does not kill us makes us stronger, more at peace, or finding peace at last.