Well, it all depends on how you look at it. Let's look at loving time/focus/commitment like food. You can't offer him a full meal that you'd be cooking together, just a snack. Still, a snack can be a wonderful, very enjoyable thing, it can help someone get through their day, and it's not like a person can't keep hunting for someone to collaborate with them on a full meal just because they've got a friend with whom they share a delicious snack. On the other hand, if they really, really, really like that snack and are super attached to the idea that food should be eaten in meal form... it could be pretty damn frustrating to have just a snack. Of course, still other people are farmers who make their own food, and don't need more than the occasional snack with others (in case you lost the thread of the analogy, I mean solo-by-choice poly people there).
So, by offering him a snack, and being clear that it's not a meal, you're in no way shortchanging him -- you don't *have* to offer him anything, after all. It's up to him to decide whether a snack of a relationship would be more delightful or more frustrating. He's not borrowing the snack from your husband -- your husband has his own meal of your time/love/energy to content himself with. What you're offering would be a different, special snack that would be just for the two of you.
I've probably taken this analogy far enough, but you get the idea.
There's one more thing to consider, actually -- maybe it doesn't have to always be a secondary relationship. People do, in fact, occasionally develop *two* primary partnerships. I'm not saying you should go into this, if you two decide to give it a try, with that expectation, by any means -- it would take time, be hard, and depend on lots of factors, like how well he gets along with your husband and kids. And it would mean eschewing the closet, at some point anyway. So, it's a long shot. But it seems like it'd be a good idea, if you haven't already, to talk to your husband, and look inside yourself, and decide whether you think that could ever be on the table. Heh, not that I meant to continue the food analogy any further (y'know, "on the table").
The major players. Me, 30ish bi female. Gia, girlfriend of 4+ years. Clay, boyfriend/dom. Davis, ex/friend/"it's complicated." Eddie, roommate & fwb.
The supporting cast. Eric, Gia's husband. Bee, Gia and Eric's toddler. Dexter, Gia's lover. Helen, Eric's lover. Izzy and Nikki, Clay's partners. Liam, Eddie's husband.