Well... there's another element to the equation, of a geometrical variety. She does kinda have a teeny-tiny-vaginy, not something I'm totally used to, and I'm not on the small side myself. If I started to go soft with my wife, I could thrust a few times, blood would start flowing again and all would be well. With this new woman, if I'm not rock solid, there's just no way I can stay in there. Condoms are a turnoff too, but certainly not the main culprit here by any means.
As for how I feel about it... She turns me on, I like her, we make each other laugh, there's definitely chemistry, and a connection. Not like the connection I have with my wife, but I honestly don't think I'm going to feel that with another human being ever, nor do I want to - that's its own special thing, and I'm absolutely ok with anything else being icing on the proverbial cake. If I thought I was doing something wrong, I wouldn't be doing it. Maybe I felt bad because my wife was alone, on the floor, in the other room, and I didn't want her to feel like she'd been pushed aside. If she had been, I dunno, at work or in bed with us, even if she wasn't partaking, maybe it'd have been a whole different equation.