Sometimes what poly does is open up our heads and hearts and shine a light on all of the stuff that's not healthy and has been hiding from us. This can be scary, and unpleasant, and the first reaction and easiest thing to do is to stop doing poly so you can shove the unhealthy stuff back into the darkness.
Or... you can see this is a golden opportunity to work on the unhealthy stuff, pull it out of the darkness, take it for a ride, see what it's all about and eventually let it go.
I think a counselor might very well be in order, if only to help you find out where the feelings/emotions are coming from and work through them.
In the end, this work will help YOU become a healthier person. In return, it will help you become a healthier partner for your wife, and help your relationship.
I'd suggest giving that a little time before making any big decisions.