Search papers, labs, and topics across Lattice.
PipeLive addresses the challenge of dynamically reconfiguring pipeline parallelism (PP) for LLM serving without service interruption, crucial for serverless platforms and heterogeneous GPU environments. It introduces a novel KV cache layout and incremental KV patching mechanism inspired by live virtual machine migration to enable live KV resizing and synchronization. Experiments show PipeLive reduces time-to-first-token by 2.5x compared to disabling KV resizing and reduces reconfiguration overhead to under 10ms, improving TTFT and TPOT by up to 54.7% and 14.7%, respectively, compared to a variant without KV patching.
Achieve near-instantaneous LLM pipeline parallelism reconfiguration – going from seconds of downtime to under 10ms – by borrowing techniques from live virtual machine migration.
Pipeline parallelism (PP) is widely used to partition layers of large language models (LLMs) across GPUs, enabling scalable inference for large models. However, existing systems rely on static PP configurations that fail to adapt to dynamic settings, such as serverless platforms and heterogeneous GPU environments. Reconfiguring PP by stopping and redeploying service incurs prohibitive downtime, so reconfiguration must instead proceed live and in place, without interrupting inference. However, live in-place PP reconfiguration is fundamentally challenging. GPUs are already saturated with model weights and KV cache, leaving little room for new layer placements and necessitating KV cache resizing, at odds with systems like vLLM that preallocate for throughput. Moreover, maintaining KV consistency during execution is difficult: stop-and-copy introduces large pauses, while background synchronization risks inconsistency as states evolve. We present PipeLive, which enables live in-place PP reconfiguration with minimal disruption. PipeLive introduces a redesigned KV cache layout together with a co-designed extension to PageAttention, forming a unified mechanism for live KV resizing. It further adopts an incremental KV patching mechanism, inspired by live virtual machine migration, to synchronize KV states between source and target configurations and identify a safe switch point. PipeLive achieves a 2.5X reduction in time-to-first-token (TTFT) without KV cache overflow compared to disabling KV resizing. Furthermore, compared to a variant without KV patching, it reduces reconfiguration overhead from seconds to under 10ms, and improves TTFT and time-per-output-token (TPOT) by up to 54.7% and 14.7%, respectively.
