Having agreed to date him? It's not a "dating idea," it's a new partner for you. If you aren't comfortable with it, knowing there will be tension with his wife and your husband, don't do it.
You haven't signed a contract
Or, you know, if you have, that's cool, too
If he's really that much of a jerk, that he would cut the friendship over it, is he really worth keeping as a friend?