There is nothing that says in the rule book of poly that you have to all be dating one another.
If you just like the guy, then date him and enjoy your metamours for what they are... metamours. They aren't quite friends, sometimes lovers, often people just quietly respect their partners partners and thats it. It seems like these other women are just a different interest to him. I would shrug it off and carry on being a good partner to him and getting on with other things in life.
As to the liking the one woman. It sounds like she isn't interested or would of shown interest. Maybe if you tell her that you want to spend more time with her she will take notice. Otherwise, ya, move on.