I admit I have a hard time understanding all of this as well but the whole story is so terribly tragic and I do feel for you.
It strikes me that you are still talking about what happens going forward as if it is your decision alone to make. I think you are taking too much of this responsibility on yourself. If this is about Matt feeling like he had lost any say over what happens with his own family then this can not be improving that problem. I can now understand why he was so quick to jump in to talk about custody battles in his thread. The issue of whether you will ever be in another relationship again may be up in the air but the issue of whether your children will share a bond with anyone else seems like his line in the sand. There really is no third option once he has made this decision.
You mention that he is begrudgingly allowing Si to see them despite having stated that he is not ok with it. This is exactly how you got here in the first place. I see the picture you are imagining. He states what he believes should happen, but as you are also their parent, he is allowing you to make the final call on whether they can see her. If this truly is his line in the sand, and you decide not to join him on that side of this decision, he will continue to feel he had no say in what happens. He could continue to say he was telling you how he felt and you ignored it. It could lead to him blowing up and perhaps asserting his say over the children the only way he thinks he can, legally.
I am proposing that rather than continue to state his opinion and then allow you to make the decision, he needs to make that decision final and say "No." It should not be your job as peacemaker to guess what is too much to try to prevent this blow up. He should decide what he can and can't handle, make a final decision accordingly and stop agreeing to let things happen that are not ok to him. This seems like it could be an issue in parenting decisions between you two whether polyamory was ever involved or not.