You're not "the bad one." The hard thing for any person, whether mono, poly, bi, straight, gay, whatever, is time. And when we meet a new love, we want to delve into that because oxytocin, the love hormone, is a powerful. And adding a new relationship is never easy. You wouldn't be saying all this unless you really cared about him and his well-being.