It does sound like an issue of control. When the two of you, or he alone, decide to have sex with another, or 2 others, he's fine. When you then take it to the next level, being with a guy who's had a long-standing crush on you, your h gets jealous and turned off.
The first scenario, to me, sounds more like swinging (focus on sex with no emotions), as hot wifery (him choosing a guy for you to be with and getting turned on by it), or wife-swapping (each of you with a member of the other couple).
The second scenario sounds more like poly. Your new guy really likes you, and vice versa, there are emotions involved, not just bodies.
Quite often there is a gender divide. The men are fine with the sex, but not the emotions. The women want the emotions as well as (perhaps) the sex. Your h might now be afraid that you'll leave him for the guy that cares about you and appreciates you. He might be expressing these fears by not wanting to touch you for several days after you are with the other guy. If he doesn't want to have sex with you for 6 days after you have a loving sexy date with the new guy, and you see the new guy once a week, you'd never have sex with your h!
At this particular point, it sounds like you are poly, but your h is a swinger. Only you two can work out how to proceed from here. Your h might be having a lot of the emotions the mono partner does in mono/poly couples. Read up on what he might be fearing by doing a tag search on this board on former "mono/poly" threads.
Love withers under constraint; its very essence is liberty. It is compatible neither with envy, jealousy or fear. It is there most pure, perfect and unlimited when its votaries live in confidence, equality and unreserve. -- Shelley
Mags (poly, F, 61) loving miss pixi (poly, F, 39) since January 2009, living together since 2013
Punk, 42, ex bf, manchild. I've been Punk'd!
"Master," (mono, 34), miss pixi's Dom for 2 years