Have you read the "5 Love Languages" book? I've found it extremely helpful when communicating with my partner about the reasons behind some of my needs. I'm a "Quality Time / Physical Touch" person, and some of what you've written about your husband seems to indicate that maybe he leans heavily in those directions as well (even if they aren't his primary languages).
My point is, if he defines love through those languages, look at what you're "telling" him - he has less time with you, and no physical intimacy. If those activities are what help him feel loved, then yes, he's going to feel like love is being withheld if those activities are "taken away".
It might be good (if you still want to remain non-physical for now) to read the book and learn what EACH OTHER'S languages are, so you can start to understand not only what each other wants, but how you both EXPRESS love.
I do have to agree with DingedHeart above - your needs WITHIN YOUR MARRIAGE may not be met at the moment, but you have other outlets for your intimacy. Your husband does not. He's CRAVING it right now, hence the desire for another GF. If he wants time and intimacy and you can't give it to him, how can this be resolved without him feeling that another GF can help? That might be a question worth exploring with him.
Mono. Divorced, two kids (DanceGirl, 15; and PokéGirl, 12), two cats, one house, many projects.
My partner. Poly.
Chops' other nesting partner, Poly. Also in a relationship with Shaggy
Chops' other other partner (heh). Married, Poly.
Poly, in a relationship with Xena
External mono/poly blog: From Baltic to Boardwalk