If you have strong feelings for this new man, but the threesome aspect of being with him and his wife makes you uncomfortable, then why not *just* be with him? Do they consider themselves a package deal?
Check out the essays here, especially the ones listed under the header Poly Configurations along the righthand side of the page:
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.