I understand. You want to feel special.
I think you can feel special even if you're not the only one (for instance, you feel special to your parents without being an only child, etc), but I can definitely see how you could feel like just a line on a chalkboard if he's had lots of other people before you.
It seems to me though, he cares about you enough to stop the rest - he didn't do it for the others, or you wouldn't have met. At some point there is a line between the effort HE has to put in to show how special you are and make you feel truly appreciated, and the amount of insecurity YOU need to get rid of. It's not easy in your situation though, things are piled up against you: you don't have much experience, so that's one thing you can feel insecure about. He's far away, another thing. He's married, another thing. He has lots of experience, another thing.
So I can totally see where you're coming from. If it's going too fast for you, I would say, tell him so and have him slow down. After all, if you are special to him, he'll want to respect you and respect your own pace, and let you set it.
When I asked about "in a few years", I did want to know what you felt about this potential scenario now. Of course, what you would actually feel, nobody can know. But putting yourself in a situation that removes the distance, the newness, the "weakness" of the relationship, and seeing if the feelings are still there, that can help you see if that's what's causing them or not.
I find this kind of exercise helpful. Forever, asking a mono wife "if you were 100% sure he'd never leave you for her or stop loving you, would it be better?" helps you know if the jealousy is tied to those feelings.