I was best friends with my first love and when we broke up we would still spend time together. THe MONO Gf that i ended up being with completely freaked out at the thought of me haging out with my ex.
This definitely created some resentment towards that lover at the time. she too is now an ex and i'm in a much healthier poly relationship where my POLy partner and i have no problems with ex-es. actually she frequently speaks to one of her exes on facebook all the time.
I know that there is a feeling (allbeit tiny) that is wierded out by it because that part of their life is over, but you cannot deny that it may have been a very significant part of your loved one's life. and to deny them to even talk can cause only trouble.
However, if there is an interest for more than talking, then you two need to talk about it to see how each of you feel about it. someone already said vetos work. and they do unless it causes them to want it more cause they can't have it.
i would say encourage talk with a temp veto and then you might start to relax more or your partner may not in fact have any interest in pursuing his ex as a lover again.