Subject
8 entries
Attention
Bookmarks
How Attention Got So Efficient: GQA, MLA, DSA explained
A YouTube explainer on how attention mechanisms evolved from full multi-head attention to the efficient variants powering modern LLMs — covering GQA (Grouped Query Attention), MLA (Multi-head Latent Attention from DeepSeek), and DSA (Dynamic Sparse Attention). Useful for understanding why modern inference is faster and cheaper than it was two years ago.
Landmark Attention: Random-Access Infinite Context
Landmark Attention paper extending LLaMA 7B to 32k token context by using landmark tokens to represent blocks of input, enabling attention-based retrieval of relevant blocks rather than attending over all tokens. Achieves near-GPT-4 context length through architectural change rather than longer pretraining.
How AI Transformers Mimic Parts of the Brain (Quanta Magazine)
Quanta Magazine's September 2022 piece on emerging research showing that transformer attention patterns converge with how the brain processes language and vision — a surprising empirical finding with implications for both AI interpretability and computational neuroscience.
Attention and Augmented Recurrent Neural Networks (Distill)
Distill's 2016 interactive visual explainer covering four augmented RNN variants — attention, memory networks, neural Turing machines, and neural programmers. A foundational reference for understanding how external memory and attention were grafted onto recurrent networks before the transformer era.
Hopfield Networks is All You Need
Ramsauer et al. introduce a modern continuous-state Hopfield network with exponential storage capacity and prove that its update rule is mathematically equivalent to transformer self-attention. This is the foundational paper connecting classical associative memory to the attention mechanism, explaining why transformers work through an energy-function lens.
How to Do Nothing
Jenny Odell's EYEO Festival essay on resisting the attention economy — arguing that doing nothing is an act of political resistance, not laziness. Later expanded into a book that became a touchstone for tech workers questioning the value of constant productivity.
The Joy of Quiet
Pico Iyer's 2012 NYT essay arguing for deliberate disconnection from digital noise — written just as smartphones were making constant connectivity the default. One of the first mainstream pieces to name what we were trading away and why retreating to silence was becoming a luxury.
Productivity Porn
Vivek Haldar's term 'productivity porn' names the trap of consuming productivity content as a substitute for actual work. The activity feels productive because it's about productivity — but it's a high-quality form of procrastination.