To me this sounds like a pattern of him just not connecting at all to your desire for communication. I don't think any of us have the tools to tell you how to fix that, and it's very hard to do poly without it. Therapy is a very good suggestion. Is it possible he could have Asperger's Syndrome? The stuff about just walking away during conversations and pretending not to hear things sounds like more than just shyness to me.
Me, 30ish bi female, been doing solo poly for roughly 5 years. Gia, Clay, and Pike, my partners. Davis, ex/friend/"it's complicated." Eric, Gia's husband. Bee, Gia and Eric's toddler.