If the 2 of you really can't come to a place where the relationship will work for both of you in a healthy way it's most likely better for the both of you to part ways on good terms so that you can both live the lives that you need to live in order to be happy. I don't know that there is any answer here that will keep everyone happy. From the sounds of it you don't want to leave your wife. Sounds like it's a tough spot to be in. Good luck
Everything will be ok in the end. If it's not ok it's not the end.