I think you will probably find a big difference in how poly men approach being with your wife, from how swinger men approach it. If the guy is experienced in practicing ethical polyamorous relationships, it's more likely he will respect boundaries and actually want to develop a relationship with her first, rather than just get her in the sack right away. Of course, there are both gentlemanly swingers and poly buffoons out there, but if you are both smart about how you try and find people to be in relationship with, I think you will be okay for the most part. Remember also that poly guys will often have their own sets of rules or boundaries they must abide by as well.
Perhaps you should check if there are any poly groups in your area and just socialize with people first before getting into dating anyone. Take it slowly.