For me in my relationships this kind of scenario you describes indicates a lack of connection. If he is not telling you about other partners and his attachment to them then I would wonder how your connection could hope to build.
I, for one, need to know emotional details about my partners love for others and what they do in their lives... not necessarily what they do sex wise, but how much they love them, what they love about them, what they think about feel about everything in their lives.
Knowing peoples emotions is huge to my sense of depth and connection and therefore makes me want to be intimate with them... this is me... what is it that turns him on? what turns you on to being connected? This is where I wonder if there is a gap in your relationship so far.
I wonder if taking a chance and making yourself more vulnerable to him and asking him to do the same would help... it sounds like you have reached a plateau in your relationship where it might all seem a bit hum drum to him. Would you be willing to go deeper? Or is the fact that he has other partners and your fear that you don't stand up to them in some way holding you back?
I'm not sure here, but I hope that some of what I say resonates with you in some way. Good luck.