I think Dirt makes a really good point.
Incidentally , I do think long distance relationships can work... but it depends what you want and what works for you. I've always liked them and made them work.
I currently live in the UK and my primary GF lives in the US! It's been like that for two years. We see each other 6 months out of the year and the rest of the time is emailing, skype and going into a game called RLC, which is like Second Life. It's not even a casual relationship - we consider each other a wife, a life partner, parents to our daughter, and are fully committed. If that level of commitment can work across the ocean, I don't see why a more casual relationship couldn't work with a 5 hour distance. But.... as I say.... you know yourself and maybe it's not for you!
For the future, or if you decide to pursue a little something with him now... there are a couple of really, really good articles that might help you.
Have you ever heard Sternberg's theory of love? It is not directly based on polyamory; but I feel it can definitely relate. The idea is that there are 7 different types, or stages, of love, based on the combination of different facets - passion, intimacy and commitment. The final, ideal, stage is 'consumate love', which is a solid combination of the three. Consumate love takes time - you aren't going to have it for other people straight away. This may help you to understand and explain what kind of love you feel for your friend, as opposed to your husband. And I do know what you mean - I've fallen in love with two people at once... but not in the same way. Using this article, I have discovered that I absolutely feel consumate love for my girlfriend and have yet to feel that level of love for a secondary partner.
Another useful article was posted by Gala Girl and it really helps to breakdown insecurity and jealousy. Pages 5 and 6 are extremely helpful in terms of giving pointers on how to deal with poly, whether you are the active partner, or the partner who is struggling:
It sounds like you and your husband have an excellent foundation to polyamory, or to whatever path you choose to follow together.
I wish you all the best of luck and hope we hear more of your story as it goes along!