Subject
4 entries
Bigscience
Bookmarks
What Language Model to Train if You Have One Million GPU Hours?
An ablation study by the BigScience group comparing architectural choices and training setups for large multilingual language models targeting 100B+ parameters within a fixed 1M A100 GPU-hour budget. It shows that careful architecture and training setup decisions at the 1.3B scale transfer predictably to larger models, making principled design tractable even at extreme scale.
BLOOM: Open Multilingual Large Language Model
BLOOM is the first open, multilingual large language model trained transparently by a global coalition of AI researchers — 176B parameters, 46 languages, trained on the Jean Zay supercomputer in France. A direct counterpoint to GPT-3's closed access.
BLOOM Training Corpus: 2D Embedding Visualization
A 2D UMAP visualization of 10 million text chunks from the BLOOM training corpus, encoded with all-distilroberta-v1. A rare window into the geometry of a frontier model's pretraining data.
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
PromptSource is an IDE and community repository for creating, sharing, and iterating on natural language prompts that map dataset examples to input-output pairs for language model training and evaluation. With over 2,000 prompts for ~170 datasets, it provided the infrastructure behind the T0 family of models and multitask prompted training research.