I've found that I only have a problem 'letting go' when I don't actually have a grip in the first place: when I have a deep emotional bond with someone and we're not in a place to recognize it, and that person is with someone else. (I had a best friend once upon a time, and a mono boyfriend, and... yeah.)
The closest I get to difficulty letting go these days is noticing the moment pass by in which I'm societally expected to have difficulty, wondering for a moment if I'm actually feeling any, not finding any problems, and carrying on with my day. Then again, I'm just as hard-wired to be poly as Mono is to be mono, so it's not "in my genes" to worry about this sort of thing.
How about everyone else?