Thanks for your thoughtful replies.
The issue is definitely with her husband since she is not seeking out other partners.
If I were having an affair, if the relationship were solely sexual, then I'd have no hang ups about her husband. But I'm emotionally invested in the relationship, as is she, and that makes all the difference.
When I'm at her house, she, her husband and I often eat dinner together. One night, he and I were alone and began talking about our respective relationships with her, but I wasn't in a place emotionally where I could comfortably talk about it. But he and I get along fine. I wouldn't call us friends, but we are on genuinely friendly terms.
I realize that if I choose to work through these frustrations, it will take quite a bit of time, which I'm willing to do if I can see some glimmer of hope that I'm actually capable of doing it. Some times that hope seems more likely than at other times.