Not sure if that's how it is for you, but I can imagine your wife's other partner prefering to keep you out of sight / out of mind in terms of the overall shape of the relationship could contribute to feelings of jealousy. Jealousy can sometimes be abated by feeling like you're as much of a concern in your wife and his relationship (in terms of being respected as a separate but sustaining relationship, that should be given good energy and support to flourish too) as his relationship with your wife enters your own domain.
Sounds like he's a good person and respects you, but mmm... wanting to maintain a fantasy world where you don't exist (even in a small way) could keep a crack in the door for the ol' green-eyed monster. Just a thought. May well be off-base here.