Subject
2 entries
Transfer Learning
Bookmarks
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Vu, Constant, Al-Rfou, Cer, and Lester (Google Research/UMass, 2022) show that initializing soft prompts from a related source task dramatically improves prompt tuning, achieving near fine-tuning performance while keeping the base model frozen. The result reveals that prompt initialization is a critical and underappreciated factor in parameter-efficient adaptation.
Pre-Trained Models: Past, Present and Future
Comprehensive survey of large-scale pre-trained models (PTMs) tracing the evolution from BERT and GPT through four research frontiers: architecture, contextual use, efficiency, and interpretability. Required reading for understanding how self-supervised pre-training became the unified backbone of modern AI.