Lack of communication. Common issue, but absolutely deadly. If he's not giving you the communication and speed that you require, then you're going to have to be the one that fronts up and starts talking to him about it.
I can't make any judgements on the situation because the two of you aren't talking about it - there seems to be no boundaries set, no limits discussed and no plan. Recipe for disaster imo, so start talking!