Subject
2 entries
Memory Optimization
Bookmarks
AirLLM: 70B LLMs on a 4GB GPU
AirLLM runs 70B LLMs on a single 4GB GPU and 405B Llama3.1 on 8GB VRAM — without quantization, distillation, or pruning. It optimizes inference memory usage itself rather than compressing the model, preserving full model quality on consumer hardware.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
ZeRO (Zero Redundancy Optimizer) eliminates memory redundancy in distributed training by partitioning optimizer states, gradients, and parameters across data-parallel processes rather than replicating them. It enables training models 8x larger than prior methods on the same hardware and forms the foundation of Microsoft's DeepSpeed library.