It's so interesting to me how women often seem to have more of a problem with their partner's emotional "affair" and men often seem to have the biggest problem with the sex. You're not alone, at any rate. I'm sure there's lots of good advice and techniques out there from other poly guys who've wrestled with this and come through it to find a place of compersion.
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.