This sounds all about the woman...? is it? Where does her husband come in? Easy does it. That is a whole lot of energy between you three and if there is none towards him there could be a whole lot of hurt.
hope that isn't true and the fun can continue without restraint.