Hmm. I don't know about that one.
It appears that you are talking more about opening your relationship and including his brother moreso than the fact that you are romantically involved with the brother and now want to include him in your original relationship.
That's not exactly poly.
And further to other's comments, where are his thoughts on this? Is it something that he and your husband have discussed and are now including you? Have they shared women in the past?
Poly can be complicated enough. Chances of it being something indefinite and long term are slim. When love becomes involved, it gets complicated. And when it ends, what would the family dynamic be with a former lover? Will it be a secret? What if the rest of the family finds out?
Oh my goodness. I got cold shivers. My bf's brother is quite attractive and very sweet, kind, generous man. But I could never imagine wanting to include him in our sex life! To me, he's family, and I don't go there with family. But that's just me.
As a "secondary", it can be tough. But every circumstance is different. Every relationship defines it's own boundaries. I don't think that anyone can really answer that part of your question for you.