first let me send you a big hug.
Your feelings are what they are and are neither 'right or wrong'. They are yours. I, too, would feel some of the same that you are.
I'm sure others will have better advice than mine but I think maybe you have to try to let it go just a little bit. I think infatuation usually runs its course and, believe me, sneaking around with a married person gets old quick. You always want more than they can give.
Do you have a trusted person you can both sit down and talk with-like a marriage counselor sort of person? Sometimes that third party can defuse and make effective communication much easier.