Each person in your life should be as significant as the relationship between the two of you balances out. (Relationships need to be roughly balanced, right?) If you're in a relationship where the significance you each give the other doesn't allow you time or energy to have anything else of significance in your life, then, yes, if you want anything else significant, you're going to have to give less significance/attention/love/time/whatever to the person who otherwise takes up all your significance/attention/love/time/whatever.
Some people have a point where they are "polysaturated" and don't have any significance/attention/love/time/whatever to give anyone else. Self-awareness of this point is helpful.
Others call this "scarcity thinking" and reject the idea that we diminish ourselves or our relationships by having more of them.
My personal opinion is our love is infinite, but our time is not. YMMV.