Why did you leave J in the first place? Have those issues been formally addressed? Or are they just no longer relevant, because you're no longer together?
People break up for a reason. It's not uncommon, after getting over the hurt from the break up, to start thinking that the relationship could just pick up where it left off. It's easy to be nostalgic for the good things and forget all the problems you had before.
And that's just about the wisdom of getting back together with an ex, in general. That doesn't even begin to address the fact that you're in a relationship now with someone who doesn't like your ex, and whom your ex doesn't like.
If you decide to pursue this, and I'm not convinced that's the best course of action, then the first step is to talk with Z about it and see if he's even remotely open to the possibility. At the end of the day, it's your life and your choice. But if he doesn't support that choice, it could be the end of your relationship with Z.
I think it's worth mentioning... I don't buy into the whole "you have no control over who you love" idea. Maintaining your feelings and desires for J is a choice. If it had been him who let you go, you'd have no choice but to figure out how to get over him. But you let him go, and you're keeping him just close enough to avoid dealing with that loss. You don't want to let him go, so you aren't taking any steps to heal that wound. Recognize that this a is self-interested choice, as it means you're asking two other people to do really hard work on themselves, just so you don't have to do really hard work on yourself. That's your choice to make, but do so consciously.
“As I am sure any cat owner will be able to tell you,
someone else putting you in a box is entirely different
from getting into a box yourself.” —bisexualbaker