This all sounds sad and depressing. I'm sorry you're going through all this....HOWEVER....My wife and I's experience was not so different.
We had been married for about 7 years when sometyhing she said rang in my ears a bit. Being an over horny and under-sexed guy, I had to jump at the opportunity. She said she thought that my best friend at the time, was hot. Well, this all led into lots of fantasy talk in the bedroom. LOTS of communication, etc. I.e. Did she REALLY think he was hot in "that" way, would she do something wiht him if she were presented with the opportunity, etc etc. Well, eventually, one night, he was over at our house, she said she was going to have something to drink. I got up to get it with her. She said that she wouldn't mind sleeping with him.
So....I took him to Taco Bell and asked him if he would be interested in going out with her and did he find her sexually atractive. He almost dropped his chin on the floor of the pick up. LOL Anyway, it eventually turned into 3 year relationship. After about 3 years, he told us one day that he was getting married and couldn't see my wife anymore.
HUH!!???!?!? We didn't even know he had a girlfriend! Well, he got married, and requested that we not mention anything about our times together with him, to his (now) wife. We respected that request, but also lost a great friend in the process. It was sad and sucked, but led us to where we are today. We have been married for 20+ years and are openly poly.
My point is this: Had she not been honest with me about her feelings, I would not have been honest about mine. It would have been SUPER disasterous.
Be slow, and confident. He has opened up to you. Respect that and don't throw it away. Be honest with him. Let him know your fears. For my wife and I, we both had fears and still do. We have found the only way to get opver them, is to simply dive in and get wet. The fear may still be there...for a while. But the longer you survive w/o anyone leaving or divorcing, the more those fears rest and subside.
He is most likely not looking to "replace" you, but rather to add to his own personality and life experience. Remember, YOUR view of poly may not be his and visa versa.
(Just like mine and my wife's doesn't match Magdlyn's or her's match ours.....but we still coexist...Right Mags?
) You need to talk with your DH and iron out some details. His absolute honesty is going to be key here...as well as yours. And COMMUNICATE!
hope this helps.