Even if you both agreed on the rate of development and the nature of your other relationships, it's not something you can realistically control on the ground anyway.
I can't say whose right or wrong here, but I am of the opinion that the whole spirit of saying to your partner that they are free to pursue other relationships is that - the freedom to pursue them. It's a contradiction to say, pursue something, but make sure it fits within these constraints.
I think your best bet is to find out exactly what fears are behind him wanting to put restrictions on your other relationships, and then work on that. Sometimes, it's necessary to pass up a relationship with someone if it's not viable for all involved. But if you are restricting your relationships because of his fear, that's another story.