I have a personal policy of making it impossible for someone to break up with me twice. I do that by refusing to get back together with anyone after we've broken up. Break-ups happen for a reason, and spring to summer is never enough time to make personality changes severe enough to merit a second chance.
The very nature of passive-aggressiveness is that people usually don't realize they're doing it. They tend to feel that they're the ones getting screwed and they're just doing "what they have to" to keep it even.
You wrote him a love letter, he said thank you and then asked why you wrote it. You responded with a cold shoulder, basically telling him to go away. That's the opposite of "I love you." If you really had no hidden agenda, wouldn't the appropriate response to his question be something like "I've noticed we've grown distant lately, and I wanted to take the first step towards getting us back on track." Your response does indicate that you expected a specific reaction, i.e. that you had a hidden agenda.