Why stay away? It sounds like adding a romantic dimension to your friendship with him would be lovely and very satisfying, not to mention fulfilling some lusty urges. I don't think that just because you had made up your mind not to be with men should mean you need to avoid any entanglements with this man. He might be perfect for you!
The world opens up... when you do.
Oh, oh, can't you see? Love is the drug for me. ~Bryan Ferry
"Love is that condition in which another person's happiness is essential to your own." ~Robert Heinlein