This is probably a great opportunity to discuss with him the level of transparency that you expect regarding any/all contact with lovers, past, present or potential. The fact that you're unhappy about this seems to me to be a red flag. And he might not even be consciously aware that he's causing you to be uncomfortable (he simply may not be thinking this through very well). What will be revealing is his reaction to you if you bring this up with him. You have a right to voice your opinion about his behavior.
Take the risk of thinking for yourself, much more happiness, truth, beauty, and wisdom will come to you that way. - C. Hitchens
Me: Male, bi, 48, flexible