Maybe you have to think of it from the other side of the fence too? For example, with Jen being the poly one in our marriage, I feel pretty stretched thin thinking of her having more than one BF. Different situation, sure, but I think it's pretty applicable.
Not to mention, I think Jen feels the same way... But that could have more to do with the time thing. While I don't think time and love go hand in hand, time is a factor no matter what, especially with the busy schedules most people have these days.