Yeah, after reading your post, it seems to me the main problem is that you prefer sex with people you have a stronger bond with. So if anything else happens, it might not feel right. Maybe you feel guilty not towards your GF but towards yourself, for doing something you would rather not have done in hindsight.
I think nycindie is on to something. Try and develop stronger bonds before letting sex happens, and see if it makes you feel better. Maybe you simply haven't been completely true to yourself.