Search papers, labs, and topics across Lattice.
The paper introduces Distributed Weight Data Parallelism (DWDP), a novel inference parallelization strategy for LLMs on multi-GPU systems like NVL72 that eliminates layer-wise inter-rank synchronization by offloading MoE weights and fetching missing experts on demand. To mitigate overheads, they propose split-weight management and asynchronous remote-weight prefetch. Evaluated with DeepSeek-R1 on GB200 NVL72, DWDP achieves an 8.8% improvement in end-to-end output TPS/GPU while maintaining comparable TPS/user performance.
Ditch the synchronization bottleneck: DWDP unlocks faster LLM inference by letting GPUs work independently, boosting throughput by 8.8% on NVL72.
Large language model (LLM) inference increasingly depends on multi-GPU execution, yet existing inference parallelization strategies require layer-wise inter-rank synchronization, making end-to-end performance sensitive to workload imbalance. We present DWDP (Distributed Weight Data Parallelism), an inference parallelization strategy that preserves data-parallel execution while offloading MoE weights across peer GPUs and fetching missing experts on demand. By removing collective inter-rank synchronization, DWDP allows each GPU to progress independently. We further address the practical overheads of this design with two optimizations for split-weight management and asynchronous remote-weight prefetch. Implemented in TensorRT-LLM and evaluated with DeepSeek-R1 on GB200 NVL72, DWDP improves end-to-end output TPS/GPU by 8.8% at comparable TPS/user in the 20-100 TPS/user serving range under 8K input sequence length and 1K output sequence length.
