I'm sorry to hear all this. It's very sad.
I wouldn't try to remain friends with J. It doesn't sound like he cares about your feelings enough to make a friendship worthwhile.
Also, he seems to have no boundaries whatsoever with his wife. Ick.
I totally get what you mean about feeling like you wasted time with someone who was ultimately not worth while. However, the great thing about polyamory is that you didn't totally waste that time--you were building / enjoying / maintaining other relationships during that time.
Single, straight, female, solo, non-monogamous.