My boyfriend, Fly, hardly ever tells me he loves me. It took maybe 3 years into our relationship for him to say it, and he almost never says it first, unless he's a little tipsy.
Every couple months I'll ask, "Hey, you love me?" and he'll say "Yes, I love you." Usually, if I tell him I love him, he says "Thanks." It used to bother me, now it makes me laugh.
He feels that the word and sentiment are overused, and have lost their power for a lot of people. He also comes from a non-demonstrative, cold (in my opinion) family, so he doesn't have a lot of practice. He also used to believe that the L-word had connotations of monogamy and exclusiveness, and he had to learn to trust my ability to be non-monogamous and allow him that freedom before he could feel comfortable saying it to me. The only person he can unreservedly say it to is his son.
Moonlight told me she loved me about 2-3 months in. I would reply with "I adore you" or "You're so special to me!" I really wasn't ready to make the sort of unspoken commitment implied by telling someone you love them, and it took me a while before I finally busted out with it. Now she and I are a regular goofest!
For me, I'm an I-Love-You sayer. It's really important to me that people I care about know it, and in my opinion, you can't trust that your actions will convey that message. So I ILY all over the place, to my family and friends and lovers, but only when it's true. It just happens that I find it easy to love!
I think it depends on so many things - personality, background (I have a very gooey family, we always say ILY on the phone or when we say goodbye), culture, associations with the word, experiences...
If it really bothers you, I think you should say something, as others have suggested. Worms or no worms, if you need the words it's going to be difficult to sustain the relationship without getting what you need from it.