Search papers, labs, and topics across Lattice.
This paper introduces ReMP, a runtime model parallelism reconfiguration framework that enables low-downtime adjustments for large language model (LLM) serving. By decoupling the model parallelism topology from runtime state, employing a two-dimensional KV cache migration mechanism, and facilitating end-to-end online reconfiguration, ReMP allows for rapid topology switches with minimal service interruption. Experiments show that ReMP can achieve reconfiguration in 1-7 seconds, offering significant performance improvements over traditional static configurations under dynamic workloads.
ReMP slashes reconfiguration downtime from minutes to mere seconds, revolutionizing how LLMs adapt to fluctuating workloads.
Current large language model (LLM) inference systems universally deploy ultra-large-scale models using a combination of Tensor Parallelism (TP) and Pipeline Parallelism (PP). However, existing systems treat the model parallelism topology as a static configuration that cannot be flexibly adjusted at runtime. This rigid design creates a fundamental contradiction with the dynamically changing inference workloads in real-world scenarios. State-of-the-art systems lack online reconfiguration capabilities and can only switch configurations by restarting the service, resulting in several minutes of service interruption, KV cache loss, and prohibitive recomputation overhead. To address this problem, this paper presents ReMP, a runtime model parallelism reconfiguration framework that supports low downtime. ReMP achieves dynamic adjustment through three key techniques: (1) decoupling the model parallelism topology from runtime state to avoid full service reconstruction; (2) designing a two-dimensional KV cache migration mechanism to preserve reusable cache states after TP/PP changes; and (3) implementing end-to-end online reconfiguration. Experiments demonstrate that ReMP can complete most topology switches within 1-7 seconds on models ranging from 7B to 70B parameters, achieving speedups of tens to over a hundred times compared to the restart approach. Moreover, ReMP significantly outperforms fixed configurations under dynamic workloads, delivering superior performance in terms of TTFT, TPOT, and output throughput.
