Just gotta pop in here to say, it sounds to be like you want two differnet things. it sounds like, he wants poly (yes?) but what you describe (casual sex, no emotions) is swinging, NOT poly. poly is about building multiple sustainable relationships, swinging is having casual sex. (i dont know if any more senior members with rebut me on this, but this is how I see it). So you two need to talk.
I find it suprising you are more jealous as the primary than you were as a secondary. I am far less jealous (not at all) of my husband dating because he and i live together, kids together etc, than my boyfriend dating because i am NOT his primary (though ive mostly gotten over this)
I think you nee to figure out what open relationship it is you two are having.
keep on keeping on