Ugh, I would not tolerate that kind of disrespect. Sounds like he expects you to accept whatever crumbs he throws your way... but you deserve a full slice of pie!
There is no typical scenario in poly, but I know if I were you, I would rather he had said, "What night is good for you? Let me know and I'll clear my schedule with my wife." Then if it doesn't work out, he comes back and says, "Friday doesn't work [for whatever reason], but I have Saturday daytime, Sunday night, or Monday night available. Which would you prefer?" YOU have a say! A considerate man -- a gentleman -- would ask you when you would like to get together even if he does have to check with his wife, and then try to accommodate you, NOT tell you to have his leftovers. How dismissive he is of you and the value of your time! Grrrrrr.
From now on, take a stance. Don't wait for him to tell you what is okay with his wifey-poo. YOU tell HIM, "If you want to see me, I have such-and-such nights free. Let me know which works for you by [specific date]. If we can't get together this week, I guess it'll be next week, then." And if he doesn't give you a satisfactory answer by your deadline, go make plans with friends, family, other lovers, but don't sit around waiting! If a couple of weeks go by and he has a fit because he hasn't seen you, he will have to start making more reasonable accommodations to be with you. This is how we train people, frankly, to start being more considerate. He has chosen to have a girlfriend and a wife, so he had better learn to manage his relationships so that both of you feel respected and cherished individually, and not treat you like some extra thing he's trying to squeeze in, after everyone else gets theirs. Ugh, treating people like this disgusts me. If the parameters of his marriage won't allow for you to have a voice, then walk - really, why stay? Most women have to get out of the habit of being too available -- being accommodating is what society has taught us, but being overly accommodating enables others to trample over us!