The thoughts you are dealing with right now can be very scary and it sounds like they're really swallowing you up. Not fun.
You feel like if he spending time with her because he would rather be with her than you. Well, he is spending time with her because he wants to spend time with her. That does not mean he does not want to spend time with you. Since you say you are his primary he is probably spending more time with you than he with his secondary. But she is his secondary, so he does want to see her. Try to take the negative aspect out of all of it and just focus on the positive.
How can you love more than one person at a time? The more I read about how the human brain works when choosing mates the more it's starting to sound like some people are just wired that way. When some people enter into a relationship with another person the part of their brain that is searching for a partner shuts off and they are content to only be with one person. However, there are many people whose brains continue to search for other people, it's not a fault of the person they are with, it's just how they're wired. Also try to look at it from a resources point of view. Love is a renewable resource, there is always more to go around of whatever type. Just because you loved boyfriends in the past does not mean that whatever love you had for them is now used up and unavailable for your current partner. It's the same for him. What is a non-renewable resource is time and so if you feel like you are not getting enough time with him then bring it up and let him know.
It sounds like you're having some negative thoughts about yourself in there too. This is where your partner could be very handy. Talk to him, get him to explain why he chose you as his primary over anyone else. What makes you unique? What is special about your relationship? I bet there will be tons of answers. Also try reading books like The Ethical Slut or Opening Up (I can't remember the authors), they often help.
You said that there is more to this so if this advice was of any use feel free to PM me and talk if you want. Hang in there and I hope things start to look up.