I know we're not voting on this or anything, and this isn't some sort of court of poly opinion, but this does sound very unhealthy to me in several ways.
It would be one thing if you were content with this arrangement, but it seems very obvious that you are not.
This has to be good for both of you, otherwise it is, by definition, one-sided. And from the way you have characterised your relationship with him, you obviously feel that it's not in terribly good shape.
I'm not sure if you have heard the old crutch about married life "well, we are having lots of problems so we're going to have a child - that'll make things better."
Well, there's a poly version that is done quite often: "Relationship broken, add more people". It very very rarely works. Most poly people, when they see the sort of situation like you have, and are being "wooed" would run a mile. If they didn't, they would just get dragged into the drama, being forced to take sides, or becoming a punching bag for both - a way to redirect their anger at each other to someone else.
My advice would be that you both forget about bringing anyone else in, and first decide whether the tow of you can have a happy, functioning relationship, even if that is in the expectation that someone may come in in the future. If you can't get to that point, then maybe you should consider going your separate ways.
"Listen, or your tongue will make you deaf." - Native American Proverb