Subject
3 entries
Long Context
Bookmarks
From Deep to Long Learning?
Stanford Hazy Research argues the next frontier is moving from deep networks to networks that can process very long sequences — motivating state space models like Mamba as a shift away from transformer attention's O(n²) complexity. A prescient 2023 post about where sequence modeling was headed.
Memorizing Transformers
ICLR 2022 spotlight proposing memory-augmented transformers that use approximate k-NN lookup into stored (key, value) pairs at inference time, without weight updates. Scales to 262K token memory with consistent perplexity improvements — an early approach to giving language models dynamic, test-time-updateable knowledge stores.
SLED: Efficient Long-Text Understanding with Short-Text Models
SLED (Sliding-Encoder and Decoder) lets you apply short-context pretrained models to arbitrarily long documents by chunking input with overlap and fusing representations in the decoder. Competitive with specialized long-context models without the expensive custom pretraining.