It scares the hell out of me. I don't want to break a heart. I don't want to damage a family.
That is not within your control. You are not the only heart on the line. You are not the only person in the polyship on the line.
So long as she and her husband are aware you exist and on the same page about her having extramarital relationships? They are practicing an open or poly model ethically?
Then they know the risk. They either have the intrapersonal skills and interpersonal skills to play ball well. Or they don't.
Some relationships go the long haul, some do not. You know the risks too. You are not a teen dating person.
At this time you seem to feel the pros outweight the cons. So... play ball. Could see how it unfolds.
People feel all sorts of things. If it comes time to part, all will be sad, and time will heal, and hopefully good memories remain because you all behaved authentically, honestly, ethically toward each other. It can end WELL.
You could talk about how you want to be as exes should that ever come to pass. Then you know what to expect from yourself and each other.
I want to be true to myself.
Then you maintain good character and make sure your behavior is rooted in good character. That you CAN control.