I'm an atheist for lack of a better term. I've heard someone toss out pantheism before, but that just strikes me as off.
I can see why some people would think that poly folks are pagan though. From the internet, it appears as if most people who are poly are also interested in wicca and other forms of spiritualism/paganism. Check out the Meetup groups and some of the more popular poly writings and you'll see how frequently that connection is true.
In my circle of friends, many of whom are also poly, I find atheism to be the most common.
The internet is a lousy way to make extrapolations about things though. Not all LSD users are Buddhists and Pagans, but browsing the net for people's experiences might tend one to believe it's true.