Sounds to me like they are working on their relationship and as of yet you have done nothing to warrant distrust or resentment from the husband. By being there you're causing two things to happen. The first is that you're forcing them to deal with the issue. It is very possible that he would just put it off if he wasn't pressured so now he has to spend time thinking about it. The second is that by being the attraction you're ensuring she doesn't go off and be with someone else without his consent. That would destroy any chance at them being poly if not their entire relationship.
You're doing more good than harm. If you slip or if it becomes apparent that he is taking a dislike to you, then you can reconsider. I say keep playing it slow.
"Regret for the things we did can be tempered by time; it is the regret for the things we did not do that is inconsolable." - Sydney Smith
Kyle: 27 year old male
Katie (rymmare): 25 year old female
Kids: girl: 5 years old, boy: 3 years old