Subject
13 entries
Fine Tuning
Bookmarks
Editing Models with Task Arithmetic
Task Arithmetic shows that fine-tuning deltas (task vectors) can be added, subtracted, and combined arithmetically to edit model behavior — without any additional training. Negation forgets a task; addition combines capabilities; analogical transfer works across tasks.
Reducing LLM Deception with Self-Other Overlap Fine-Tuning
Self-Other Overlap (SOO) fine-tuning reduces deceptive behavior in LLMs by aligning internal activations for self-referential and other-referential prompts — cutting deceptive responses from 73% to 17% on Mistral-7B with minimal capability loss. A representation-level approach rather than behavioral supervision.
Kiln Fine-Tuning LLM Models Guide
Kiln's practical fine-tuning guide walks through generating synthetic training data (the 'Ladder' strategy: 10 manual → 30 LLM → 1000 synthetic) and running training jobs on OpenAI, Fireworks, and Unsloth. Emphasizes data quality over cost during generation.
Fine-tuning language models: a practical overview
A clear introductory overview of LLM fine-tuning from the GenAI Guidebook — covers why and when to fine-tune, the mechanics of weight updates, and the major techniques including LoRA and RLHF.
LLM Course: Roadmaps and Notebooks for Large Language Models
A comprehensive GitHub repository by Maxime Labonne with roadmaps, notebooks, and articles for learning about LLMs — from fundamentals through fine-tuning and deployment. One of the most starred LLM learning resources, covering the full practitioner pipeline.
QLoRA Minimal: Fine-tuning Notebook
A minimal Jupyter notebook demonstrating QLoRA fine-tuning — strips away framework boilerplate to show the core mechanics of 4-bit quantized LoRA training. Good reference for understanding what QLoRA actually does at the implementation level.
Alpaca LoRA: Instruct-Tuning LLaMA on Consumer Hardware
Alpaca LoRA applies Low-Rank Adaptation to fine-tune LLaMA on consumer GPUs — making instruction-following fine-tuning accessible without data center hardware. Part of the March 2023 democratization wave that showed open LLM development didn't require massive compute.
Training Language Models with Natural Language Feedback
Proposes learning from natural language feedback on model outputs rather than simple comparison labels, using a generate-filter-finetune loop to train GPT-3 to human-level summarization with only 100 feedback samples. Natural language carries more alignment signal per human evaluation than pairwise comparisons.
Scaling Instruction-Finetuned Language Models
The FLAN-T5 and Flan-PaLM paper (2022) showing that instruction finetuning across 1.8K tasks improves performance on zero-shot, few-shot, and chain-of-thought prompting across multiple model families. Established instruction finetuning as a general-purpose method and released Flan-T5 checkpoints that shaped the open-source LLM ecosystem.
Large Language Models Can Self-Improve
Huang et al. (2022) show that LLMs can bootstrap their own reasoning ability by generating chain-of-thought rationales, filtering with self-consistency majority vote, and finetuning on the high-confidence outputs — no human labels needed. A clean demonstration that LLMs can improve themselves without supervised signal.
Textual Inversion for Stable Diffusion
A patch enabling textual inversion in Stable Diffusion — the technique of learning a new text token that represents a custom concept (a person, style, or object) from just 3-5 example images. Textual inversion was the first practical method for personalizing Stable Diffusion without full fine-tuning.
Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
OpenAI's InstructGPT paper (2022) shows that a 1.3B model fine-tuned with RLHF on human preference data is preferred over raw GPT-3 at 175B — establishing that alignment via human feedback is more important than raw scale for following instructions. This is the foundational paper behind ChatGPT and the instruction-tuned era.
Differentiable Prompt Makes Pre-trained Language Models Better Few-Shot Learners
DifferentiablePrompt (DPT) replaces discrete token prompts with optimized continuous embeddings, enabling gradient-based prompt tuning for few-shot learning. Published at ICLR 2022, it established that soft prompts can match full fine-tuning with far fewer parameters.