You make it sound as if having sex with these other people just happens to you. "It felt so natural... we just wound up doing it, what's wrong with that?" Just thinking sexual thoughts about an old friend doesn't need to mean that you will follow up on those thoughts. "I didn't think I desired anyone else, but then I did!" Whoops! Attractions don't always need to be acted upon sexually. I am sure you know that.
There is nothing wrong with casual sex if you are comfortable with that, but you make it sound like breaking your agreements occurs of their own accord, without any forethought by you. But there was a choice you made to have sex with these other people, and a choice you made to break your promises to your boyfriend. You can also choose not to have sex and to keep your promises. It's rather silly to pretend that you have no responsibility to your relationship, agreements, and what you do with your body -- don't you think?
It certainly seems like what you really want is just a loose arrangement for you to have casual sex with others, not really polyamory. Are you telling yourself you love your friend that you had sex with just so you can call what you're doing "polyamory?" Or is it really love? It sounded like it's a fairly recent thing, so it's a bit unclear.
It also seems that you and your boyfriend might have different ways of defining what you want. It would be beneficial for you to discuss everything in detail, and when one of you makes a statement about what you want or need, you both need to keep asking questions and to get clear on what your agreements mean.
Good luck & welcome.