Welcome to our forum.
It sounds like the first thing you need to do is ask him if he finds you attractive. But I don't think you messed up or anything. Romance is always awkward in the beginning.
If he does find you attractive, will you be able to accept that he has a wife, and that he has an open relationship with her? Perhaps you and his wife should meet up and see if you can strike up a friendship. Just a possibility.
Because his relationship with his wife is open, it should be okay (in theory) for you and him to pursue something. But find out what their rules and boudaries are -- if there's anything to pursue here.
Remember that true love always has a risk of heartache attached. And remember that honest, considerate communication is the highest priority in polyamory.
I hope everything goes well for you. Just take it in little steps.
Love means never having to say, "Put down that meat cleaver!"