Search papers, labs, and topics across Lattice.
This paper introduces LUMEN, a fault-tolerant system for large language model (LLM) serving that optimizes recovery from worker failures by coordinating load-aware decisions across checkpoint placement, interrupted-request distribution, and serving capacity restoration. By addressing the inefficiencies of existing systems that either restart requests from scratch or rely on fixed neighboring worker checkpoints, LUMEN enhances the overall performance of LLM clusters under failure conditions. The evaluation shows that LUMEN significantly reduces both serving and recovery times, making it a robust solution for maintaining service continuity in distributed LLM environments.
LUMEN slashes recovery times in distributed LLM serving by intelligently coordinating load-aware recovery strategies during worker failures.
Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.
