So he has to choose at her behest? Either her or you? And you are hurting because it was supposed to be swinging only and not love and don't like seeing how torn he is over this request?
Well, could accept it became love. Could also be glad your spouse's feelings for you too must run very deep for him to be this upset over it. Feelings cannot be controlled at will. But behavior can be controlled. So if participating in swinging led to this -- could not engage in swinging anymore in future. If he stays with you, are you going to renegotiate your marriage boundaries to take swinging OFF the table?
You both are going through the stages of grief.
Maybe skipped "shock and denial" stage since you both knew it was an Open thing and knew the risk of swing turning to something else. But there's still the rest to get through.
Thing is, you are mourning different things.
You are mourning swing turning to love and perhaps scared/jealous stuff clouding the view. Perhaps fearing he will not choose you.
He is mourning the end of a polyship and having to choose.
MOVING IT FORWARD
Could tell spouse you see he hurts terribly. Could tell spouse you know it is hard and are sorry he hurts. Maybe show him the stages of grief article and ask if he needs to see a counselor and (assuming you are willing to keep on tending to him as the wife) arrange an appt for him if he cannot manage to do it for himself.
(Could arrange one for yourself.)
Could tell spouse you want to be with him. This is your willingness.
Could tell your spouse you know that for the marriage to exist BOTH partners must be willing.
Could tell spouse you will expect him to tell you where his willingness lies -- with you or with her. And that you will accept the answer he gives and even if his willingness lies with her, you will try to make it smooth for the kids and yourselves with a smooth split/divorce. But in the meanwhile, you are willing to try to keep your shared home life as normal as possible to give him that peace/space.
Then could wait.
You can do no more than give him respect, dignity and space while he sorts himself out if you want to be with him. If you don't want to be with him any more that's another set of options but it sounds like you do want to be with him.
You may want reassure from him, want comfort from him -- want lots of things in your own grief process. But at this time he may not have it to give
because he's so run down to empty himself. So again I encourage you to lean on someone else you can talk to -- a friend, a relative, a counselor for the meanwhile.
Ask him out on a date -- spend time together if this is what you want.
Could not sweat his emotional process. Or if it does become unbearable for you to witness and he's treating you badly/neglecting you? Ask him to move out. Or you move out. Then he can feel what he would be missing at home and you set a deadline for him making a final call how it is to be.
Kinda have to know if you are willing to take the "let him arrive when he arrives" approach or "I need to know by X so I can plan out my own life here" approach. You guys are there living it -- could talk about what approach this is here.
Neither of you can live in limbo forever. In that sense the other lady has it more together with what she wants/needs/ and where her limits/boundaries are. You could have your own wants, needs, and personal limitations/boundaries articulated clearly here. So could your spouse with his.
When people have those laid out clearly, it makes hard calls easy to make -- you just look at what lines up and what doesn't. It might still FEEL just as hard to do, but the actual doing is a lot more clear -- that lines up, that does not. There. Sorted. Done. Best decision for the situation is ____ then. YKWIM?
Hang in there,