IMO, you're not in a poly relationship. Your husband is simply having his affair in your home rather than playing away.
As a couple the two of you should have negotiated who wanted what and how each other (or another partner) would provide that. Poly relationships are negotiated set-ups where everyone involved works to ensure everyone else is ok and needs are being met. He sounds very selfish.
Have you talked to him about how you feel?