Sometimes it takes mini steps, backtracks and even reboots (I am a geek what do you want)...in the end (here comes a cliche) there are other fish in the sea and as you progress everyone involved will ideally get more comfortable and confident.
Sometimes its takes a few tries to make it work, not everyone is meant to be with their firsts.
I think sometimes people forget when they dated, how many girls did you date before you found the one. Add the complication of a wife and it might put it into perspective. Poly is complicated and doesn't always work. As heartbreaking as that sounds....its just reality.
Best of luck with the continued journey.