If your energy is driven to your primary partner, I think that is where it should be, and that your primary partner needs to either respect this, and give you the majority of his energy and time right now, or that he needs to take control and tell you how it will be (not sure of your relationship dynamic in this regard, of course).
If he does the first, then it will be more up to you to progress things, which is as it should be (in a traditional sense), since you are the one having the hardest time with it. Since you 'want' it to work, I think you could likely get there. But it has to be more on your ability, and speed, to do so.
in luce vive, vive in amore