Good points, Jericka and others. I do get that I'm not obligated to pursue anything with M. The reasons I wanted to give him a chance despite not being sure of my attraction to him were these:
1. I am a very visual person and I did not want to dismiss what might be a wonderful human being just because I didn't find him particularly handsome to look at. People can grow on me, and our first date was only for a half hour, and we only hugged goodbye. While most of my partners have been more or less conventionally attractive, one of the few loves of my life would have been called downright homely by some.
2. I find that I rarely meet men who seem truly open to love and emotional commitment, even in the poly community. M. seemed like he would be one of the rare few.
3. I did like the idea of exploring tantra a little more with an experienced person. (In fact, I'm probably going to continue taking his girlfriend's workshop.) I just did not want it to define our relationship, up front. I didn't want goals. I wanted to go on a few dates with him and see how we did, before we got into all that.
I think I am going to take the advice I've received here and tell M. that I just want to be friends.