Well, yeah, that is a double standard.
It's now up to you to decide how to handle things. If you do nothing to change the situation, you will be tacitly endorsing him having another partner while expecting you to not have one.
This issue is important enough that he has to address it. If he keeps trying to weasel out of that conversation, you have to evaluate whether or not the relationship with him is worth keeping. As it is, it appears he wants to play freely while keeping you on a leash.
I suspect he suffers from an underlying insecurity as to whether you'll want him if you're also involved with somebody else. That you'll compare him with the other guy and decide you like the other guy better. You may want to ease his mind on that issue prior to pinning him down for the discussion on who can play with whom and so on.