I was married to a woman for a time. She is still a huge love in my life. The love didn't end, our paths together did. We are still close.
I met my now husband the way you describe. I now live with him and another man and we raise our child together. I have another boyfriend and a girlfriend as well. Things change, growth happens, people come and go, but it isn't necessary to "get rid" of people because their journey is going on a different path. That is the beauty of poly relationships... they can last beyond the standard "we are together" "we are not together." Its all a matter of creating what works for everyone and actively loving/supporting/caring/living together for as long as you see it is beneficial.
I started talking to J (my then wife) before talking to PN about opening our marriage up as I felt I was non-monogamous. Her and I discussed what it could all mean, what we could gain, what we could lose. She agreed to me talking to him about it and I did one night with her there. He was in agreement that we should see how it went and we tried being a triad for a time. It turned out my wife was going along with it all for me and was pushing herself to be something she is not. On the night we all ended up in bed together she broke down and could not go forward with anything sexual so she said she would leave while we did... It wasn't a good idea. She was really distraught and angry afterwards. I don't even remember if PN and I actually had sex. I just remember the pain I had caused. It was baffling to me as she had agreed! Yet was visibly in more pain than I had ever seen her in before.
The anticipation of new love can over shadow reality more times than not. NRE (new relationship energy) is not the truth. Its an illusion. Respecting that as you go along and pacing yourself is what I recommend... talking about it as a first step is actually the easy part... what follows can never be the same. I learned that that night. I was in la la land and was not following my gut and not seeing how I was causing someone pain.
I would make sure you know exactly what you are giving up as there is no turning back once the "poly talk" has happened. He could just be fine as a good friend... or maybe not. It could be the best thing that ever happened to both of you or a disaster. The thing is that if you are even here asking how to talk about it, you have likely chosen to take the leap. Good luck!