I'm a newbie since July to the concept of poly but the term incestuous would be "blood" relations having intimate encounters. You and his brother are not blood related so there is zero incest happening.
The dynamics of your relationships may seem strange to the outside world but in my opinion, it seems no different than any other V, only your men are brothers.
I don't know what to tell you about your husband's wishiwashness except to stand your ground and hopefully he'll work out his issues with the relationship. Goodluck.