I have always acted in a monogamous way-and then cheated. But my cheating was never linked strictly to a sexual interest. I would simply see something in a person that attracted me to them in a mental and emotional way and, because I never felt I could be honest with my partner, it led to cheating. So I can attest to polyamory being less about the sex and more about other connections. If he were really just looking for a way to screw around he wouldn't tell you about it, nor would he wish for you to meet this other woman or seek your own other relationships.
All that being said, I understand also the fear of the sexual side of things. My hubby and I were poly in the beginning and went into a mono relationship, then back to poly with me being the one with a girlfriend. They didn't have a sexual relationship (though they had in the past) but were best friends, affectionate, loving, and it may have headed down that path had the relationship lasted. Once I became comfortable with being poly and with the emotions behind the sex I started to become willing to open up that part of it, something I never thought I would do.
It is hard to change your ideas about what a relationship "should" be. But you don't have to. Like any other relationship, you can choose to remain in it or not. You can choose how deeply you want to be in this lifestyle. It's a good thing he's up front with you from the beginning as honesty is most important. Just take it slow, communicate to him clearly your fears and doubts, and don't be afraid to ask him questions to calm them. There's a poster on here-MonoVP-who is a mono man with a poly woman who can probably give you alot of feedback on your situation. Good luck.