Ok. No need for apologizing for lack of clarity in your state of mind. It's understandable. I just thought it would be easier to ask so we could better relate to/advise/offer support.
First, (and I should have said this before) this situation just plain sucks and I am so sorry you are left hurting. I can't imagine how I'd feel if my husband pulled this on me but it wouldn't be good. He acted selfishly and I'm sure your heart is breaking, years of love before or not. In fact, probably more so because of the history.
Before I address him, though, I would like to address the woman in this situation. This group DOES sound like swingers and not truly poly people. When you and your husband entered the party and she flirted with him and casually touched his genitals in front of you (even worse if either or both were naked) and neither he nor you reacted negatively, I'm sure she saw it as a green light. If she is a swinger, you were attending the party, she touched him intimately with no repercussions, and you then left them alone (and he did not tell her of your feelings) she really didn't do anything wrong. Not looking for an emotional attachment and not having been chastised for her attention to him and being left to her devices, she reacted to the situation she believed she was getting herself into. I still believe you should talk to her, not as an enemy, but as a fellow victim of his cheating. Make it clear he is not allowed to be with her again, nor ANYONE else while you work out your relationship. If you do this in a non-offensive way she may actually help keep others from being involved with him as well as stay away herself.
Now, for him. I can give you the perspective of someone who has been a cheater. A couple of years ago my husband and I went through quite a bit of drama, both external and internal to our relationship. We ended up to the point where we were alternating between indifferent and outright hostile towards eachother. At the worst of it I had a male friend whom I trusted and confided in. He made it clear that he had deeper feelings for me. I did not want to be involved with this other man. But, having no attachment to my husband and not wanting to lose the one I had to the other man, I cheated on my husband. It was entirely out of selfish need for the affection and attention I felt I wasn't getting. I soon realized I had nothing left to lose when we started talking divorce and I confessed. For the first time in years my husband and I really talked. For the first time in years I saw emotions in him other than resentment, anger, and indifference. We realized there was still love there and it was worth working at. About two years later and we are happier and more stable than we have ever been. But it took effort and desire on both parties to make that happen. And time. Your husband may have any reason for going through this period in his life. But if he is willing to control himself, communicate, and seriously work on the issues and you are willing to give him that chance and work yourself, this sad experience can be a catalyst for a major change for the better. But BOTH of you have to be willing to make this work, so continue to not be a doormat please. The differences in relationship philosophy seriously worry me as well as your suspicion he just wants to party without caring about the consequences. He needs to remember you have a relationship and what he does affects you. So you get a say in it. Marriage is not a dictatorship-no relationship should be. If he can't understand that...I'm sorry, but my only advice is to let him go destroy himself if he must, but do not go down that path with him.
I wish you well and hope everything works towards what will ultimately make you happiest.
I'd rather be hated for who I am than loved for who I am not.