If your outside connections just involve sleeping with other people, you have an open relationship. For it to be poly, you'd have to have actual romantic relationships with other people. From what you've said, it's unclear whether your partner wants a poly arrangement or an open arrangement.
That said, the issues for you are likely much the same--you're dealing with jealousy. One of the most essential processes involved with polyamory is to figure out exactly what it is you need from a relationship; dealing with jealousy actually helps sort that sort of thing out. Jealousy isn't necessarily a bad thing--it's a signal to attend to what you need.
So, pull out your jealousy and dissect it. What, exactly, is it composed of? What drives you to feel jealous? Are you afraid he doesn't really love you? Are you afraid he finds somebody else more exciting?
What do you fear will happen when he's with somebody else?
What do you fear will NOT happen when he's with somebody else?
When you've got a list of fears, then you can figure out how to get those addressed. You sit down with him and share your fears and negotiate ways to address those so that you get what you need while he also gets what he needs.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.