Before I put in my two cents, I needed some clarification. Are you practicing POLYAMORY or SWINGING? In polyamory there is a deeper connection than sex. It involves multiple loves who share in your life as well as your bed. The attachments are primarily emotional and the sex, while tied in, is often secondary. Swinging is more of a relationship open to outside sex partners but not deep emotional attachments. You may be casual friends, but usually not deeper than that.
The party you described sounded like a swingers party, especially being this woman was touching your husband's genitals and this was somehow not a crossing of a boundary. Also you describe her as not available to a relationship with your husband.
I'm confused as to where on this spectrum you fall. Can you explain what you meant when you said you were polyamorus?