Maybe you could start something with your ex again? Did you tell her you still love her? Your wife didn't have to make the call for both of you. It seems most of this issue is due to no one expressing their feelings and needs until the last moment or not at all. There is no need to stay in something that isn't working for years. Life is too darn short for that. If you love her and your wife, then tell them and see if something can't be worked out where you can have a "Vee."
As to it all feeling wrong; I wonder if that is because you didn't know anyone else. You aren't alone and it isn't wrong. Just not the norm.
Anyone want to be friends on Facebook?
Send me your name via PM