Subject
6 entries
Distributed Training
Bookmarks
Colossal-AI: Open-Source ChatGPT Training Replication
Colossal-AI released an open-source implementation of the ChatGPT training process (SFT + RLHF) that runs on a single GPU with 1.6GB memory — 7.73x faster than naive implementations. Made the ChatGPT training pipeline accessible to researchers without multi-GPU clusters.
Alpa: Automated Distributed Training for Large Models
Alpa is a system for automatically parallelizing large neural network training across distributed hardware — finding optimal parallelism strategies without manual configuration. From a Berkeley/CMU research collaboration, it targets the challenge of scaling models beyond single-GPU memory.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
ZeRO (Zero Redundancy Optimizer) eliminates memory redundancy in distributed training by partitioning optimizer states, gradients, and parameters across data-parallel processes rather than replicating them. It enables training models 8x larger than prior methods on the same hardware and forms the foundation of Microsoft's DeepSpeed library.
Colossal-AI: Distributed Deep Learning System
Colossal-AI is an open-source distributed deep learning framework that makes training very large models more accessible — cutting GPU memory requirements by up to 10x versus standard PyTorch. One of several systems research projects responding to the GPU memory wall problem in 2022.
Near-Linear Scaling of Gigantic Model Training on AWS
Amazon Science's post on achieving near-linear scaling when training gigantic language models across thousands of GPUs on AWS infrastructure. Covers parallelism strategies (tensor, pipeline, data) that make training 100B+ parameter models economically feasible.
Swarm Training
Shawn Presser's Swarm Training explores distributed ML training across many commodity machines with low-bandwidth interconnects — democratizing large model training beyond clusters with expensive NVLink. Part of the broader open-source effort to train large models outside of big lab infrastructure.