I just read through this from the beginning and it seems that you've already done what I was going to suggest. Have a relationship with him on your terms rather than waiting on him to decide what those terms are going to be. A friendship where sex may or may not be on the table at any particular time sounds very healthy.
Everything will be ok in the end. If it's not ok it's not the end.