I have all sorts of snarky stuff I want to say about this guy, especially as you seem to spend a lot of time thinking you want to break up with him but instead I'll just suggest you say something along this line to him. (BEFORE you get there, so he has time to let it sink in)
"I really like you, and you've seen I am not opposed to group sex, but I want to be the one to bring it up if it is going to happen. If you show me you respect this boundary of mine, I will feel much more comfortable exploring this with you in the future."
Happiness will never come to those who fail to appreciate what they already have.