I would say try to let it go. This presumably attractive guy might legitimately be into you, but he has bee involved with your. That alone is enough reason to shy away from it.
I do believe pursuing it could potentially hurt your relationship with your daughter. What if this guy just cuts her off with no warning and picks up with you? People always say, "The golden rule of FWB is no feelings," but sometimes feelings just happen.
I agree with Mags. You can probably successfully work through it. I have no experience with this type of thing. I do know sharing a guy with my sister was off-limits. I would have never dated one of her ex boyfriends. So my mum? That would have been a resounding no for me as well.
I hope you work it out, though.