Guilt is a tough one for me too. Here is a thread I started on it... http://www.polyamory.com/forum/showthread.php?t=604
I don't feel guilty about leaving PN behind so much because he has worked hard at figuring out some really great things that he can do that he likes to do alone, or that I don't enjoy. He has created a relationship with himself that is strong because Mono has come into our lives. He has the best "be your own primary" relationship of anyone I know in fact! I see that opportunity as a gift from me actually. I have freed him of some of the burden (said with tongue in cheek) of having to hang out with me... he is free to hang out with himself!