I don't know, I think you can have an "affair" without it being of detriment to your partner (except for the whole dishonesty and trust breaking thing). I would define it as passion with another person that's not sanctioned and/or known about by your partner. What would passion look like? Sentiments like I love you, I want to be near to you all the time, it blows my mind the way you understand me, I feel so connected to you, I never want to lose this.
To me, a good measure of a relationship is intimacy. Physical intimacy is, well... I don't really need to explain, do I?
Emotional intimacy is getting to know someone deeply, sharing things with them that you wouldn't share with just anyone, etc. So, if you're developing that with someone else behind your partner's back... there you go.