Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces InstantInfer, a novel cold-start system for large language models (LLMs) that leverages the Communicating Finite Automata (CFA) abstraction to optimize cross-component interactions during initialization. By enabling safe concurrent execution of components without altering the original sequential structure, InstantInfer achieves significant performance improvements, with experimental results showing up to a 7.2 times speedup in cold start times across various GPUs and workloads. The proposed framework not only enhances efficiency but also ensures correctness, addressing a critical bottleneck in LLM inference services.
Cold starts for LLMs can be accelerated by over 7 times without sacrificing correctness, thanks to a novel framework that enables safe concurrent execution.
Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number of fine-grained I/O requests issued by complex software components. Although refactoring the program can yield advantages such as concurrent execution and I/O merging, this approach is error-prone and carries correctness risks when dealing with massive, heterogeneous components. We propose the Communicating Finite Automata (CFA) abstraction to systematically analyze cross-component optimization opportunities, and design a programming framework to enable CFA-based component program refactoring. This framework preserves the original sequential program structure while enabling safe concurrent component execution. We prove the correctness of the program refactoring. We apply the CFA abstraction and framework to refactor process tree creation, tensor loading, and model switching in vLLM, forming a new cold-start system named InstantInfer. Extensive experiments demonstrate that InstantInfer substantially accelerates LLM cold starts (achieving up to 7.2 times speedup) and exhibits robustness across diverse GPUs, workloads, and scales.