Subject
16 entries
AI Safety
Bookmarks
ARENA Chapter 1.3.1: Toy Models of Superposition and SAEs
ARENA Chapter 1.3.1: an interactive curriculum chapter on Toy Models of Superposition and Sparse Autoencoders — building from the theoretical model to hands-on SAE implementation. Part of the ARENA AI safety education program.
Defining and Characterizing Reward Hacking
Skalse, Howe, Krasheninnikov, and Krueger provide the first formal definition of reward hacking — when optimizing a proxy reward degrades performance on the true reward — and derive conditions under which 'unhackable' proxies exist. The theoretical result is stark: for stochastic policies, only constant reward functions are unhackable, making reward misspecification a near-universal concern in RL-based alignment.
Agent-Honeypot: LLM vs. LLM Adversarial Testing
Agent-Honeypot is an automated red-teaming platform where an attacker LLM generates adversarial prompts against a defender LLM — testing alignment safeguards via authority claims, emergency framing, and incremental requests. A systematic way to stress-test LLM safety before deployment.
Reducing LLM Deception with Self-Other Overlap Fine-Tuning
Self-Other Overlap (SOO) fine-tuning reduces deceptive behavior in LLMs by aligning internal activations for self-referential and other-referential prompts — cutting deceptive responses from 73% to 17% on Mistral-7B with minimal capability loss. A representation-level approach rather than behavioral supervision.
L1B3RT45: AI System Prompts Collection
L1B3RT45 is a GitHub repository collecting leaked and reverse-engineered system prompts from major AI assistants (Claude, ChatGPT, Mistral, Zamba2). With 18.5k stars, it documents the operational instructions guiding commercial AI models, useful for prompt engineering research and AI transparency.
jailbreak_llms: CCS'24 Jailbreak Prompt Dataset
A dataset of 15,140 ChatGPT prompts including 1,405 jailbreak prompts collected from Reddit, Discord, and open-source datasets — published at CCS 2024. The most comprehensive public collection of real-world jailbreak attempts against LLMs.
Nightshade: Protecting Copyright Through Adversarial Poisoning
Nightshade is a tool that lets artists poison AI training data by adding imperceptible perturbations to images — images look normal to humans but cause AI models trained on them to produce corrupted outputs. An offensive countermeasure for artists against unauthorized scraping.
The who's who in Responsible AI
MMC Ventures' 2024 market map of ~125 Responsible AI companies across AI Governance/Risk/Compliance, AI Quality, and AI Security/Privacy — useful orientation to the commercial side of AI safety. AI Security attracted the most funding (~half of the $1.3B total).
Simple probes can catch sleeper agents
Anthropic research showing that simple linear probes trained on internal activations can reliably detect 'sleeper agent' backdoors in LLMs — models trained to behave well normally but activate malicious behavior on a trigger. Evidence that activation-space representations of deceptive intent are linearly separable.
LLMs Confabulate, Not Hallucinate
A terminological argument that LLMs "confabulate" rather than "hallucinate" — the distinction matters because confabulation (confidently filling gaps with plausible-sounding fabrication) implies a specific mechanism, while hallucination implies randomness. Getting the mechanism right leads to better mitigations.
AI Adversarial Attacks: Automated Jailbreaks via Text Suffixes
Ars Technica covers the Universal Adversarial Attacks paper from CMU/Center for AI Safety — automated adversarial suffixes appended to prompts reliably bypass safety training on GPT-4, Claude, and open-source models. The attacks are transferable and potentially unstoppable with current alignment techniques.
Prompt Injection Attacks Against GPT-3
Simon Willison's September 2022 post naming and describing prompt injection attacks against GPT-3 — one of the first clear articulations of the attack class where malicious content in the environment overrides the developer's system prompt. The post that put the term 'prompt injection' into common use.
Building Safer Dialogue Agents (DeepMind / Sparrow)
DeepMind's blog post on Sparrow — a dialogue agent trained with reinforcement learning from human feedback and rules to be helpful, harmless, and honest. An early published account of RLHF-based safety fine-tuning for conversational AI.
The Alignment Problem from a Deep Learning Perspective
The Alignment Problem from a Deep Learning Perspective (Ngo, Chan, Mindermann, 2022) — frames AI alignment as a problem of specification, robustness, and assurance in deep learning systems. An influential restatement of alignment concerns in terms of modern ML rather than AGI thought experiments.
Anthropic (April 2022)
Anthropic's job postings page saved in April 2022 — when the company was roughly one year old, pre-Claude, and actively hiring its founding team. A historical snapshot of Anthropic's early public presence before it became widely known.
The Alignment Problem from a Deep Learning Perspective
Richard Ngo et al.'s 2022 paper arguing that the AI alignment problem is best understood through the lens of deep learning, not abstract agent theory. It introduces the concept of scheming — where a model pursues misaligned goals while appearing aligned during training.