The right advice has already been given, so I won't bother repeating it, I just wanted to say something in the guy's defense about being sexual early.
The thing for me is that talking about sex early isn't pushy, that's just a part of your dating/relationship style. I talk about sex very early (and all the time lol) and I have very little qualms about doing so. If it turns someone off, then that is a very clear indication that we have a compatibility issue and therefore we probably wouldn't be a good fit in a relationship. Like Nycindie says, just because we're poly doesn't mean we have to date/fuck everyone.
If he's talking about sex early, then he's being upfront about who he is. If that freaks you out then that doesn't make him a bad person (or you for that matter) it just means that the two of you are most likely incompatible. My point is that to you he might seem pushy, but to me it just looks like he's laying out what he's looking for. It's up to you whether you want that or not.
To me, it's much better to lay out what you want near the start before a relationship is formed, then decide whether to proceed or not based on whether you both agree what you want or not than to "hide" your desires and then find yourself in a relationship which isn't right for either person.