Well, it seems to me that your future is set for a few years. Time to raise a baby. That is the first priority I think.
You have lots of time. I have known one of my on again off again loves for 15 years. In that time we had one night in a bed together and nothing happened (pre poly), I dated his later wife for a bit (pre their relationship), they got married the same year we did, we all had kids together and then later when she and he separated we had some fun times together all three of us (husband, he and I), then he met someone and is monogamous for the time being while his divorce goes through, and he settles in with his new love. I didn't plan any of that and I had no pre-conceived notion that any of that would happen... not in my wildest dreams.
This all started at an age younger than you and now we are all in our 40's... well not my husband, he is much younger, early 30's.
You just don't know what will happen, but you do know you are going to be a dad and that is a very important role. It is the most important right now and until such time as your child becomes more independent... at around 3 they start that... at 5 they are attached and firm in who to rely on and trust. They need this to grow into emotionally healthy adults.
This is not the time to be thinking about other loves and about how to fit poly into your life I don't think. My husband and I took a break for 4 years and now have come back to it (4 years now), more mature, more grounded, more settled and sure of what we need and what future we see. We needed that time and we needed to be parents first.
I'm sorry, this is likely not what you want to hear, but if you want to do the right thing, then I suggest you buckle down and get at the task of being daddy. The other woman/man will wait until later. They will still be there if it is meant to be.... keep in touch as you are now and wait. You will soon be out of the "missing" feeling I think if you give it a chance. You can hold them in your heart and enjoy the memories. Put it all on the back burner and be patient and be a good dad.