Mlsys 2022

Pathways: Asynchronous Distributed Dataflow for ML

Google's Pathways is a single-controller orchestration layer for ML accelerators that runs sharded asynchronous dataflow across thousands of TPUs while matching SPMD performance. Designed to break the MPI-style lockstep model so heterogeneous workloads like MoE, pipelining, and foundation-model multiplexing become first-class.