I'd need more info to advise. First of all, how long have you been with your h? Have you always been discussing having an open relationship, or is it fairly new? If he only wants to swing, and you want emotional connection/love and sex, that is a big disconnect. It can take years to open a relationship properly and learn to deal with jealousy.
I've always been poly and I still have twinges of jealousy, envy, uneasiness when my bf dates-- he's had 4 new people in his life in the past year and it's exhausting me trying to keep up!
Try getting the books Ethical Slut and Opening Up for ideas on how to cope. I'm sorry you felt compelled to give up your gf, that must hurt a lot.
Love withers under constraint; its very essence is liberty. It is compatible neither with envy, jealousy or fear. It is there most pure, perfect and unlimited when its votaries live in confidence, equality and unreserve. -- Shelley
Mags, F, 60, poly-dating, loving and living with
miss pixi, F, 38
Punk, 41, M (dating since Oct 2015)
and a few more casual relationships