This appears to be a classic case of varied levels of interest. Tom wants only a casual connection while you want something quite different. It has nothing to do with being poly--that sort of conflict arises in mono relationships, too.
As for the folks who purport to be poly and yet avoid open, honest communication, well, that behavior serves as warning that you don't want to get involved with them. Seriously, there are lots of folks who claim to be poly simply to provide cover for their bad behavior. We have a recent thread on that sort of thing over in General Discussions.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.