I agree this arrangement seems a bit unfair, if you like men and women you should be able to be with either. But, if it's okay with you, then you do what is right for your relationship. It seems to me that you should be able to find your own girl, though. What are the chances you will fall for who he is with? I think that is quite a gamble and it is really limiting to you.
39 years old, married (almost) 17 years to Avatar, in quad relationship for 2 years with Busbuddha and his wife, 3 kids between us. Have been practicing non-monogamy for 8-ish years.