Perhaps you simply are not cut out for "casual." I'm not. I don't think I could share one of the most intimate physical experiences with someone and not form some sort of special feelings for the person. But perhaps that is because I need emotional/mental intimacy to find someone else attractive in the first place. Sure I recognize when someone is physically attractive ~ both men and women ~ but to me it's more like recognizing a particularly nice piece of art. If they can't do anything for my mind, then I don't want them in my body.
Now, I realize that not everyone operates the way I do, I simply mention my viewpoint / experience so that you can determine whether you relate to this in some way. You may need to re-think your poly strategy.
Additionally, you might want to check out this link: http://www.reuniting.info/resources/exchange_of_the_day
Certain behaviors tend to stimulate bonding, whether we intend them to or not.