Outside of the whole poly issue, your relationship with Y has many of the dynamics my relationship with my first husband had. And I blamed myself....and talked...and talked...and begged...and pleaded for him to go to counseling with me after it all had gone on for a long time. I acknowledged to him that I was the one who had changed in our relationship as I was coming to finally value and acknowledge my emotional needs but that I was no longer willing to settle for the relatively logical, rational, intellectually compatible relationship/marriage that we had.
What it ultimately came down to was that he had problems with true emotional intimacy/connectivity. Obviously I did, too, when we first met and eventually married. I had a very angry, emotionally abusive father, so when I met my first husband his calm, rational approach to everything seemed so "loving" or at least emotionally "safe". It wasn't until I went into individual therapy that I started to acknowledge and treasure my emotions. Prior to this I often felt I must be "too emotionally needy" and would periodically back off from expressing my wants/needs to my husband out of guilt. My husband had a severe emotional wounding as a child related to a birth defect that led peers to teasing him alot. At some point rather than go into a rage, he shut down most of the "feelings" part of his personality. He ultimately refused to go to counseling with me.
Along the way I met a guy through a business relationship who was emotionally expressive, complementary towards me, etc. I didn't consciously go looking for it...but we ended up having an affair. It solved NOTHING. Only made my life more complicated and miserable. My husband found out. I didn't deny it. THEN he wanted to go to counseling. We did, but he still NEVER changed in terms of becoming more emotionally vulnerable, open and sharing. We had two children and I felt guilty that I would even consider divorce because he really was a "good" man and father, and how could I be "selfish" and "wreck everyone else's life" because of my "neediness"??!! I subsequently starting "shutting down", developed chronic fatigue syndrome, and became very depressed, with suicidal thoughts/intent, ending up in me choosing to hospitalize myself before I acted on my plans to kill myself with the handgun I had bought.
After this I developed the courage to divorce him because my very survival depended on it. "Right" or "Wrong", "Good" or "Bad" I was who I was and needed and deserved to have my emotional needs met. It was one of the hardest things I've ever done because I hated hurting my husband and our children. But I could no longer afford to hurt myself or I would be there for NO one.
Sounds to me that Y may have some problems with true intimacy...deep from the heart connectivity. He obviously doesn't lack sexual desire because he certainly directs it multiple times per week towards his porn habit. Sexual release with no need to connect emotionally.
I can only speak from my experience....but I strongly support you on your journey of giving yourself permission to have your feelings, wants, needs honored and met by your husband. If he can't do it, then yes....he may end up hurt if you decide to divorce him. But, you have given him many opportunities to make other choices, and he hasn't. This has hurt you. (I do think the book on Love Languages can be helpful, but I'm not hearing that Y is very committed to learning to meet your wants/needs on anything other than his terms and definition of such.) Please take good care of yourself through this process whatever your choices and decisions. (And having an affair with E or anyone is only a temporary "fix" which creates more pain in the long run....She says from experience!