Yeah, the time issue can be difficult, although the benefits of having so much love in your life far outweighs that in my own experience.
The downside, for me anyway, is the way other people view my choice to be poly. Most have been supportive, or at least not openly judgmental. However, it's been heartbreaking to lose relationships with family over it. People can be so threatened by things that are 'different' that they aren't willing to even try to understand. That kind of intolerance is hurtful and very hard to come to terms with.