Subject
7 entries
Alignment
Bookmarks
Defining and Characterizing Reward Hacking
Skalse, Howe, Krasheninnikov, and Krueger provide the first formal definition of reward hacking — when optimizing a proxy reward degrades performance on the true reward — and derive conditions under which 'unhackable' proxies exist. The theoretical result is stark: for stochastic policies, only constant reward functions are unhackable, making reward misspecification a near-universal concern in RL-based alignment.
Reducing LLM Deception with Self-Other Overlap Fine-Tuning
Self-Other Overlap (SOO) fine-tuning reduces deceptive behavior in LLMs by aligning internal activations for self-referential and other-referential prompts — cutting deceptive responses from 73% to 17% on Mistral-7B with minimal capability loss. A representation-level approach rather than behavioral supervision.
Training Language Models with Natural Language Feedback
Proposes learning from natural language feedback on model outputs rather than simple comparison labels, using a generate-filter-finetune loop to train GPT-3 to human-level summarization with only 100 feedback samples. Natural language carries more alignment signal per human evaluation than pairwise comparisons.
Reward Model Ensembles Help Mitigate Overoptimization
Coste et al. (2022) show that ensembling multiple reward models substantially reduces reward overoptimization during RLHF, where a policy learns to exploit artifacts in a single reward model rather than truly improving. This is a practical mitigation for one of the central failure modes in aligning language models with human preferences.
Building Safer Dialogue Agents (DeepMind / Sparrow)
DeepMind's blog post on Sparrow — a dialogue agent trained with reinforcement learning from human feedback and rules to be helpful, harmless, and honest. An early published account of RLHF-based safety fine-tuning for conversational AI.
Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
OpenAI's InstructGPT paper (2022) shows that a 1.3B model fine-tuned with RLHF on human preference data is preferred over raw GPT-3 at 175B — establishing that alignment via human feedback is more important than raw scale for following instructions. This is the foundational paper behind ChatGPT and the instruction-tuned era.
The Alignment Problem from a Deep Learning Perspective
Richard Ngo et al.'s 2022 paper arguing that the AI alignment problem is best understood through the lens of deep learning, not abstract agent theory. It introduces the concept of scheming — where a model pursues misaligned goals while appearing aligned during training.