Not that I have the kind of experience (poly chops?) of many of the people on here, but I think every bit of the above information is right on. I think a really significant point the other folks have made is setting a time frame. It's a fine idea to put any real decision making off for the time being to look into both of your feelings, but you definitely need to have some time limit to that (one month, six months, whatever). Even if you decide "let's take two months to look into this, and then we'll figure out the next step is", you've at least given yourself a deadline. I like literally setting dates and times to discuss those issues when things get hard (which also helps you organize your thoughts and avoid making big statements when you're feeling blue).
The problem with the vague "let's take a while to think/talk about this" is that you can potentially put it off forever - especially when you're dealing with these serious (and totally understandable) insecurities. That's not helping anybody. At first it might take the stress off, but it's only going to amplify your dread and worry in the long run. And, worse case scenario, it might breed resentment or distance in him when the months go by and the issue seems to keep getting pushed away. Having set goals was the most helpful step in my entering the poly world. If you have a real gameplan, it's much easier to maintain trust, focus, and compassion.