Have you asked him how to help him through rough spots? Seriously, it seems so many people avoid this simple process.
The thing is, it's always best to ask about that when he's *not* in a rough spot. In the midst of feeling poorly, he may not feel like engaging in that sort of discussion, despite needing the comfort right then.
I've told my wife--multiple times--exactly what helps when I'm feeling down. She's never bothered to actually pay attention so she can remember it when I've been knocked down by life and can use a hand in getting back up. She'll then come 'round when I'm down and ask what she can do to help...um, not a good time to ask that. It says, in effect, that she really doesn't care as it's never been important enough for her to pay attention to--which adds to the psychological trauma.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.