Search papers, labs, and topics across Lattice.
This paper introduces KV-Pipe, a novel stage-aware KV-sharing mechanism designed to enhance the efficiency of pipeline parallelism in large language model training. By strategically converting attention layers to cross-layer KV sharing, KV-Pipe optimizes the FLOPs Imbalance Ratio (FIR) and significantly improves both utilization and throughput during training, achieving up to 9.2% higher training memory footprint utilization (MFU) and a 9.8% reduction in iteration time. Additionally, this approach benefits inference by minimizing KV-cache growth and redundant work, leading to improved decoding throughput for long-context tasks.
KV-Pipe transforms KV reuse into a powerful tool for balancing pipeline workloads, yielding significant efficiency gains in both training and inference.
Pipeline parallelism (PP) is widely used to scale large language model (LLM) training, but its efficiency is often limited by stage imbalance and pipeline bubbles. Meanwhile, cross-layer KV sharing has primarily been studied as a mechanism for reducing KV-cache costs during inference, without examining how KV reuse reshapes pipeline workloads. We present \textbf{KV-Pipe}, a stage-aware KV-sharing mechanism that turns KV reuse into a pipeline-balancing control knob. KV-Pipe starts from the tail stage, converts selected attention layers to cross-layer KV sharing in a tail-first order, and iteratively retargets the current bottleneck to drive the FLOPs Imbalance Ratio (FIR) toward $1$. The procedure is performed offline and requires only a pipeline partition and per-layer FLOPs estimates, introducing negligible runtime overhead and requiring no online tuning. Across multiple pipeline-parallel configurations, KV-Pipe consistently improves utilization and throughput, achieving up to \textbf{9.2\%} higher training MFU and up to a \textbf{9.8\%} reduction in iteration time, with larger gains at higher pipeline-parallel degrees where stage imbalance is amplified. Furthermore, the same KV-sharing mechanism provides an inference-side benefit by reducing KV-cache growth and redundant KV projection work, resulting in higher decoding throughput for long-context workloads. These results identify KV layout as a system--architecture degree of freedom for jointly improving pipeline-parallel training efficiency and long-context inference.
