Language Models

PAL: Program-aided Language Models

PAL prompts LLMs to generate Python programs as intermediate reasoning steps, then delegates actual computation to an interpreter, outperforming chain-of-thought on math and symbolic tasks. It neatly separates what LLMs are good at (translating problems to code) from what they're bad at (arithmetic).

Mapping Language Models to Grounded Representations

An NLP paper from late 2022 on mapping language model representations to grounded meaning — likely examining whether and how LLM internal representations correspond to real-world referents or structured semantic formalisms. Addresses the 'grounding problem' at the heart of debates about whether LLMs understand language.

ROOTS Search Tool — BigScience

A Hugging Face Space for searching ROOTS — the massive multilingual dataset used to train BLOOM, the BigScience open LLM. Lets researchers trace which training documents a model might have learned from.

STaR: Bootstrapping Reasoning With Reasoning

Memory-Based Model Editing at Scale

ICML 2022 paper on memory-based model editing at scale — a method for locally updating a neural network's factual knowledge without full retraining. Uses a cache of explicit edits that override the base model's predictions for targeted inputs.

GPT-3 and Arithmetic (MIT LINGO Lab)

MIT LINGO Lab's analysis of GPT-3's arithmetic abilities — probing how and when it succeeds or fails at basic math, and what this reveals about how language models represent numerical reasoning. Relevant to understanding the difference between pattern matching and genuine computation.

PEER: A Collaborative Language Model

PEER is a language model trained to collaboratively write text with humans — drafting, suggesting edits, following instructions, and explaining its changes. An early formalization of human-AI collaborative writing that anticipates modern writing assistant workflows.

PEER: A Collaborative Language Model

PEER is a language model trained to simulate the full collaborative writing workflow — drafting, suggesting edits, explaining changes, and integrating feedback — by learning from Wikipedia edit histories. It achieves strong performance on editing and revision tasks without task-specific training.

Training Language Models to Follow Instructions with Human Feedback (InstructGPT)

OpenAI's InstructGPT paper (2022) shows that a 1.3B model fine-tuned with RLHF on human preference data is preferred over raw GPT-3 at 175B — establishing that alignment via human feedback is more important than raw scale for following instructions. This is the foundational paper behind ChatGPT and the instruction-tuned era.

Pre-Trained Models: Past, Present and Future

Comprehensive survey of large-scale pre-trained models (PTMs) tracing the evolution from BERT and GPT through four research frontiers: architecture, contextual use, efficiency, and interpretability. Required reading for understanding how self-supervised pre-training became the unified backbone of modern AI.

Unifying Language Learning Paradigms (UL2)

Google Research's UL2 paper proposes Mixture-of-Denoisers (MoD), a unified pre-training objective that combines span corruption, prefix LM, and causal LM into a single framework — and separates architecture choices from pre-training objectives, which were previously conflated. The insight that objective and architecture are orthogonal opened the door to mixing paradigms that were previously treated as distinct camps.

Modeling Vocabulary for Big Code Machine Learning

An empirical study of vocabulary modeling decisions for machine learning systems on source code, evaluated across 14,436 projects. It matters because the choices made when tokenizing and preprocessing code vocabularies have an outsized impact on neural language model accuracy, yet were poorly documented before this work.

Flamingo: A Visual Language Model for Few-Shot Learning

DeepMind's Flamingo (2022) bridges a frozen vision encoder and a frozen large language model with cross-attention layers, enabling powerful few-shot vision-language capabilities without retraining either component. It set new few-shot records on image captioning and VQA benchmarks by treating visual inputs as just another type of context for an LM.

TextSynth Playground

TextSynth is a web playground and API for running large language models efficiently — built by Fabrice Bellard (of ffmpeg and QuickJS fame) using his GGML-based inference engine. In 2022 it was one of the few accessible ways to experiment with GPT-J, GPT-NeoX, and similar open models.

Why GPT-3 Matters

Leo Gao's early analysis of why GPT-3 is qualitatively different from prior language models — written in May 2020 when GPT-3 was first announced. One of the clearer technical takes on what the scaling breakthrough meant, from someone who later worked on it.