You need to tell him that you aren't comfortable with poly anymore, or for now. It wouldn't be the first time a poly couple took a break from it to handle important life circumstances. Talking to him would support your well-being, and his. Come clean!
The world opens up... when you do.
Oh, oh, can't you see? Love is the drug for me. ~Bryan Ferry
"Love is that condition in which another person's happiness is essential to your own." ~Robert Heinlein