Re (from desire
"What is the right and fair thing by both of us now? Is there something like that? He is clear what he wants, he wants both of us ..."
Well, I don't know so much what would be right/fair by her, since I don't know of her situation as well as I do yours. I think that on your part, he has moved too fast in his relationship with her and should probably slow it down. Perhaps he should even end it with her, if him being with her is just not something you can live with and be happy. But he's kind of "led her along," and so breaking up with her might not be fair to her. Did she know he had a wife (you) when she first got involved with him? Was she aware that this open/poly thing might be something that you would struggle with?
Plus he should show (each of) you that you are wanted (by him) in a way that you can emotionally relate to. For whatever causes/reasons, he's getting the message through intellectually, but not emotionally. What's your primary "love language?" Touch? Service? Gifts? Quality Time? Words of Affirmation? Whatever it is, he should be speaking to you in that "language."
It's actually be normal for you to be having some feelings of loss. Monogamy is presented to us as something of a "happily ever after dream," and throughout childhood (and adulthood), this message is reinforced, so that we start "dreaming that monogamous dream." Then, when a theretofore-monogamous relationship opens up (and becomes non-monogamous), your (involuntary) conditioning can cause you to feel like the "happily ever after" and "dream" has been lost. This is something you are probably grieving. As grief is a process, you shouldn't beat yourself up over not being able to "just get over it." You have to go through the steps of grief over time.
What else can be done to help the situation? Would setting a goal to "move back in together" in the same house/town be realistic or a good idea? What else would you need to feel comfortable with this situation?