FullofLove1052, I wonder if you might not be in a subconscious mourning phase for losing your relationship with Si? Every time you try to have a little of the old times with Si, there's Matt, afterwards, with his angry words and expressions, and you realize (again) that you can't have the life with Si that you used to. Si keeps getting pushed further and further onto the back burner. Now she will only have electronic contact with you (texts and that). It looks as if your relationship with Si is dying. I would expect you to be in a period of mourning about that.
I feel for all three of you in this situation. I can see how you would all three feel rejected, left out, etc. ... There must be some guilt at continually giving Si less and less of you. I share your sense of mourning in that sense. You are truly between a rock and a hard place.
Love means never having to say, "Put down that meat cleaver!"