What you describe is poly. Poly does not require that everybody love each other. Take, for example, tangles of three persons. Those can be triads, where each person is involved with both of the other two. Those can also be vees, where one person is involved with each of the other two and those two are not involved with each other.
If you are involved with more than one person romantically, that is poly. Your husband doesn't have to be involved with any of your other partners for it to be so. Nor would you have to be involved with any of his other partners for it to be poly.