Poly is possible, sure... it sounds like boyfriend is not interested thought so really the real issue to me sounds like how daddy is going to be involved in your and babies life... skip the sex and love part, a child is coming and organization ahead of time in terms of time management and expectations and assumptions of roles need to be figured out I would think... at least talked about.
It sounds like boyfriend is feeling threatened and if that is the case, I would start with that. Find out why and what he wants out of all this, if anything.
Anyone want to be friends on Facebook?
Send me your name via PM