Actually I do get this, Derby. When my kids were younger we lived in a rural area and I had no access to babysitters pretty much their entire childhoods. When my husband would go away to visit his girlfriend overseas for months, and I was pretty much a full-time nanny and dog-sitter with nothing but online contact to keep me from going insane (ok, I went sort of insane). :P
My suggestion is to take as much advantage as you can of opportunities to go out both with your kids and to use sitters when you can get them. The reality is that he's able to spend more free time than you are right now because of how you've structured your work and lives, and while envying him is natural, it won't help you feel any better. It might also be worth figuring out if some of your feelings come from a desire to do something different in your life that might be unrelated to your relationship or your kids.
I suggest making him aware of how you feel and trying to figure out at least some small steps towards him helping you with this when he can. Are there ways you can both restructure your lives to make things easier on you?
I also suggest trying to get some of the fun to come to you -- can you have people over more often to keep you company so that you don't have to leave your place and the kids?
Finally, I suggest not getting down on yourself about this. All parents with young kids, in my experience, feel this way at least some of the time -- they DO cramp our style and when a partner has a lot more freedom it's pretty rough. It doesn't last forever, but at the moment it feels pretty unfair and lonely.
Oh, and also, it's ok to let your partner and even your kids know that Mommy sometimes needs her own alone/playtime to feel happy. Everyone does -- kids get that, they really do -- and the happier you are, the happier your family will be, too. It *does* get easier.
Hm. I see a need for a poly-daycare support network...