Welcome to our forum.
Based on the information given, I have to conclude that you will be taking a gamble no matter which road you take. Until you get to know this man's wife better, you have no way of knowing how she will react to her husband being emotionally involved with you, if she is accustomed to swinging and that's what she's comfortable with. On the other hand, if you withdraw from this man, you could end up spending the rest of your life wondering, "What if I had given it a try?"
Perhaps you should explore this relationship a little further and see what happens. If the wife disapproves and you have to withdraw later, you might suffer some heartbreak, but without the risk of heartbreak there can't be any romance (poly or otherwise).
Feel free to look around our site and see if you find any experience/wisdom that helps you make a decision. You can always post further questions, concerns, etc.
You might find our Life stories and blogs board
to be useful and interesting for reading (or writing your own blog). The Poly Relationships Corner
might also be a good place to post.
Hope you find the information you need.