It is about having cake and eating too, but as far as I'm concerned, I have two pieces of cake, and it's VERY important to me to make sure both my husband and my boyfriend know they are adored, so that my husband especially (since we're new to polyamory) never feels like you do.
You asked how poly relationships work; well, here's how mine does:
I love my husband. Nothing with that changed when I met my boyfriend. That's how I knew I wanted a polyamorous situation, that I didn't want to sacrifice my amazing marriage, but I felt very strongly that I needed to explore this connection I had with the man who is now my boyfriend. Because I was familiar through poly friends, the notion that I could love more than one person completely was not new to me, and I communicated with my husband openly from the beginning when feelings started happening. But my husband's been an active participant in this process and in the decision to open up our marriage.
It's okay to not be comfortable with this arrangement. A lot of people aren't. It's okay to be monogamous with a polyamorous partner, but you need to be honest with yourself about what you want and need. It sounds like this is no surprise, but the fact that it's still hurting you after four years is a problem in your relationship that you and your boyfriend need to address, and if you're not alright with him being poly, you need to be honest about that and let him go.
But I will tell you that I am more in love with my husband now, after 7 years, than I've ever been, and my new relationship with my boyfriend doesn't diminish that. You need to trust your boyfriend to not just tell you he loves you, but to make you feel loved. If that's not happening, this will never work. And if you can't be with someone who's poly, that's okay, but you need to be honest about that, for both your sakes.
Good luck to you.