I'd agree. Ask him out. If he's not interested, he'll let you know (hopefully graciously). If he's interested, you guys can take it from there, address the concerns and move on, be it together or seperately, but I think it's won't be too big of an issue.
Far less than the wondering is, certainly.
Connection is the path to passion. Passion is not the path to connection.