As previously said, it sounds like you are swinging.
In swinging, the emphasis is on the physical aspect of the relationship and no love needs to be involved.
Polyamory is more about the emotional aspect of a relationship and in some cases doesn't involve sex.
They overlap to some extent, but I would say your case is into swing territory. You might want to look up "swinging" or "swinger" and see if you find something that fits your situation.
Me: 30F, straight
Seamus: My husband, 32M, straight
Dragon: My boyfriend, 29M, pansexual, married to Fox
Fox: My boyfriend, 29M, homoflexible, married to Dragon