Well you obviously aren't a co primary
Your boyfriend issues are not relevant to your duty as a parent. Who cares who you love, right now your kids relationship with their dad is threatened. Breaking up with your kids dad might help that situatiom so you break up and arrange formal visitation
That's putting your kid first.
I've been a parent for 7 years and I'm a midwife. You often become a better parent when you stop focusing on romance & relationships.