I'm involved with a guy outside my marriage, and I text and call him whenever I feel like it, whether my husband's around or not. We don't really have a time when it's "okay" for other partners to call. So I think it sort of depends on the couple's specific rules, and this guy is the best person to ask about that sort of thing.
One word of caution: if you can only call certain hours of the day when his wife's not around, and you never meet the wife, do you know if he's telling the truth when he says he's poly? Make sure he's not "just" cheating.