Search papers, labs, and topics across Lattice.
Piper is a novel distributed training system that allows users to define and implement complex parallelism strategies without the need for manual low-level coding. By decoupling the strategy from the runtime implementation, Piper utilizes a unified intermediate representation (IR) to compile execution plans that adapt to various parallelism strategies, including advanced methods like DeepSeek-V3's DualPipe. The system achieves performance parity with established strategies such as ZeRO while also enhancing memory efficiency through optimized scheduling of compute and communication tasks.
Piper enables seamless integration of cutting-edge parallelism strategies, achieving performance parity with existing methods while unlocking new efficiency gains.
Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation. Piper allows users to declare a comprehensive distributed training strategy with a small set of model annotations and scheduling directives. Each directive applies a transformation on Piper's intermediate representation (IR), a unified global training DAG that represents all computation and communication. Using this IR, Piper compiles per-device execution plans and executes them with a distributed runtime agnostic to the strategy. We show that the combined system maintains performance parity on commonly available strategies such as ZeRO, while also enabling additional performance and memory efficiency gains through joint scheduling of compute and communication in composed parallelism strategies such as DeepSeek-V3's DualPipe.
UW