For me one of the main differences between swinging and poly are the romantic feelings.
Do either of you have romantic feelings towards A? If so, it's leaning towards poly more than swinging BUT for it to be the ethical nonmonogamy that most of us prefer polyamory to be he would have to have the support of his wife in this. Since he doesn't, it would still be polyamory in the sense of having more than one romantic love, but it wouldn't be a polyamorous situation in the sense of having an ethical relationship consisting of more than 2 people.
That's all I can say in the matter, because I refuse to help someone through a problem that arises by helping someone cheat. One of the few things that I am very adamant about.