Subject
12 entries
Reinforcement Learning
Bookmarks
Defining and Characterizing Reward Hacking
Skalse, Howe, Krasheninnikov, and Krueger provide the first formal definition of reward hacking — when optimizing a proxy reward degrades performance on the true reward — and derive conditions under which 'unhackable' proxies exist. The theoretical result is stark: for stochastic policies, only constant reward functions are unhackable, making reward misspecification a near-universal concern in RL-based alignment.
Mathematical Foundations of Reinforcement Learning
Mathematical Foundations of Reinforcement Learning by Shiyu Zhao is an open-access textbook covering RL theory rigorously — Bellman equations, value functions, policy gradient methods — with a mathematical depth missing from most applied RL courses. Free on GitHub.
Genie: Generative Interactive Environments
Genie is a Google DeepMind foundation world model that generates playable, action-controllable interactive environments from a single image prompt — photo, sketch, or AI-generated. Trained on unlabeled internet videos without action annotations, it discovers a transferable latent action vocabulary.
Improving Multimodal Interactive Agents with RLHF
The Interactive Agents Team at DeepMind (arXiv:2211.11602, 2022) applies RLHF to agents that must understand language instructions and act in visual environments, using human preference feedback to train a reward model for PPO-based RL. The result demonstrates that RLHF substantially improves instruction-following in embodied multimodal settings beyond what supervised learning alone achieves.
In-Context Reinforcement Learning with Algorithm Distillation
Algorithm Distillation trains a causal transformer on sequences of RL learning histories so the model can improve its policy entirely in-context without gradient updates. A key step toward meta-learning agents that get better at RL through experience rather than parameter updates.
A Generalist Agent
Presents Gato, a single transformer that acts as a multi-modal, multi-task, multi-embodiment generalist policy across 600+ tasks including Atari, robotic manipulation, image captioning, and dialogue using identical weights. Demonstrates that scaling language model principles to non-text domains produces surprisingly capable generalist behavior.
Reinforcement Learning: An Introduction
Sutton and Barto's canonical RL textbook (2nd ed., MIT Press 2018) — the field's foundational reference, covering MDPs through deep RL in a unified framework. Essential reading before any other RL material; everything else builds on the ideas introduced here.
Algorithms for Decision Making (2-column edition)
A two-column or second-edition variant of the MIT Press textbook by Mykel Kochenderfer covering decision-making under uncertainty, from MDPs and POMDPs through reinforcement learning and multi-agent systems. This copy predates the primary 2023-01 vault entry and may represent an earlier draft or reformatted version.
A Practical Guide to Multi-Objective Reinforcement Learning and Planning
A Springer survey on multi-objective reinforcement learning and planning — covers scalarization, Pareto-based methods, and utility-based approaches for agents that must balance competing rewards. Useful reference for RL research where single-reward framing is inadequate.
Decision Transformer: Reinforcement Learning via Sequence Modeling
Decision Transformer recasts offline reinforcement learning as a conditional sequence modeling problem, using a causally masked Transformer to generate actions conditioned on desired return, past states, and actions. It matches or exceeds model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door without any value function or policy gradient computation.
Machine Learning in Finance: From Theory to Practice
Springer 2020 textbook by Dixon, Halperin, and Bilokon bridging ML theory and quantitative finance practice — covering supervised learning, NLP for financial texts, RL for trading, and deep learning for derivatives pricing. The most rigorous academic treatment of ML applied to finance.
Cogment Verse: Human-in-the-Loop Reinforcement Learning
Cogment Verse is an SDK for training and validating AI agents in human-in-the-loop learning (HITL) and multi-agent reinforcement learning environments. Provides a web UI for human participation in training alongside standard RL algorithms like A2C and PPO.