To me this sounds like a pattern of him just not connecting at all to your desire for communication. I don't think any of us have the tools to tell you how to fix that, and it's very hard to do poly without it. Therapy is a very good suggestion. Is it possible he could have Asperger's Syndrome? The stuff about just walking away during conversations and pretending not to hear things sounds like more than just shyness to me.
The major players. Me, 30ish bi female. Gia, girlfriend of 4+ years. Clay, boyfriend/dom. Davis, ex/friend/"it's complicated." Eddie, roommate & fwb.
The supporting cast. Eric, Gia's husband. Bee, Gia and Eric's toddler. Dexter, Gia's lover. Helen, Eric's lover. Izzy and Nikki, Clay's partners. Liam, Eddie's husband.