Subject
4 entries
Large Language Models
Bookmarks
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
The BLOOM paper introducing a 176-billion parameter open-access multilingual language model trained by the BigScience collaborative on 46 natural languages and 13 programming languages. It demonstrated that a community-organized research effort could produce a frontier-scale LLM without proprietary infrastructure.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
ZeRO (Zero Redundancy Optimizer) eliminates memory redundancy in distributed training by partitioning optimizer states, gradients, and parameters across data-parallel processes rather than replicating them. It enables training models 8x larger than prior methods on the same hardware and forms the foundation of Microsoft's DeepSpeed library.
Near-Linear Scaling of Gigantic Model Training on AWS
Amazon Science's post on achieving near-linear scaling when training gigantic language models across thousands of GPUs on AWS infrastructure. Covers parallelism strategies (tensor, pipeline, data) that make training 100B+ parameter models economically feasible.
How Sber Built ruDALL-E — Interview with Sergei Markov
Serokell's interview with Sergei Markov of SberDevices about building ruDALL-E — a 12B parameter Russian-language text-to-image model. Covers the engineering and research challenges of training massive multimodal models, plus the open-source culture argument in ML.