Subject
2 entries
Tpu
Bookmarks
Pathways: Asynchronous Distributed Dataflow for ML
Google's Pathways is a single-controller orchestration layer for ML accelerators that runs sharded asynchronous dataflow across thousands of TPUs while matching SPMD performance. Designed to break the MPI-style lockstep model so heterogeneous workloads like MoE, pipelining, and foundation-model multiplexing become first-class.
Swarm Training
Shawn Presser's Swarm Training explores distributed ML training across many commodity machines with low-bandwidth interconnects — democratizing large model training beyond clusters with expensive NVLink. Part of the broader open-source effort to train large models outside of big lab infrastructure.