Do you think that he feels that you're only interested in this lifestyle as a means to get back at him for his infidelity? Just a thought. I also ask because like you said, he wasn't scared to do that 3 years ago, but is hesitant on losing you now. What's changed?
As far as the life you're suggesting, I would imagine anything is possible so long as all parties involved are open and honest with each other. My questions is more of, is that what you really want? Do you want to just love your bf and be intimate with someone else? Wouldn't you rather have both with both? The key is that you all have to be happy with the arrangements whatever they may be.
I also think that if he's not completely comfortable with the situation yet, that you shouldn't be with other people yet. (Just my opinion, it's completely up to you to determie what's best for you) If you both aren't on the same page, then he might begin to build some resentment towards you because you're doing something that he's not totally happy with. Even though he's saying that he doesn't want to lose you so he'll deal with it. You don't want him just "dealing" with it, you want him to understand and be a part of it.
Let us know how things work out!
Live life to the fullest 'cause you never know if you're gonna wake up tomorrow!