So he wants a triad, while you and his wife really want a V relationship, where you and he are friends but not lovers. He is obviously pretty sexually attracted to you, but you not to him. This is indeed a delicate situation.
Some serious introspection is going to have to happen for all you, but especially you and him. Is he ever going to be happy just being friends with you, or is that going to not be doable for him long-term? Would you ever be happy being sexual with him? If neither of you think you can make that compromise, I can't see this situation ever improving. Hubby will continue to resent you for being sexual with his wife but not him. You'll continue to resent having to be on guard from his perviness and in general feel quite uncomfortable around him. Neither makes a live-in situation very fun.
Could you guys do a V without you actually living with them? Could this be a possible compromise?