I think you might be underestimating how hard it is to maintain one relationship with a small child, much less start new ones. I can't even express how hard it is to maintain intimacy with a baby in the house. If I had to deal with all of the poly issues that I've encountered over the past year while my children were babies...oh boy. Single mom, anyone? I hope you guys will give yourselves that space.
As for the rest of it...well, I'm just as interested in the answer as you are. I'm polyfidelitous, and I have no desire for casual sex or anything without an emotional bond. My husband is more casual but not really interested in trying to find anything outside of what he has. My boyfriend might not be my boyfriend because he can't decide if he's monogamous. His wife, our girlfriend, doesn't equate sex with emotions and we are currently trying very hard to make our views mesh. When we began the relationship, we negotiated (what I thought was) permanent polyfidelity, but here we are a year later renegotiating. It might just be a situation of trial and error compromises to find out what works.