Wow guys, thanks for all this feedback, I've been offline all day and was blown away by the thoughtful feedback. Especially the person who has identified with the "free-spirit" type persona.
For some reason it helps me to know that there are other people like J around and maybe we're not imagining things.
There is validity in everything you've said. I also think the advice that RP gave on another post called part-time poly is very relevant here.
It's interesting that we generally have great communication but over this I'm actually scared to talk to him. He's been much better today; I don't think he's called her and I don't even want to mention her in case it brings back his pain. But when/if he gets the blues again I will be definitely be using anotherbo's "relax, let her be...." line.
We both need to be strong, he needs to be strong without her and I need to be strong without him (his usual loving self), while he struggles with this.