Search papers, labs, and topics across Lattice.
This paper addresses the challenge of efficiently serving LLM inference in serverless environments with limited and heterogeneous GPU resources by proposing a dynamic pipeline reconfiguration approach. The method minimizes service downtime and performance degradation during online adjustments of pipeline configurations to adapt to changing workloads. Experiments on NVIDIA A100 and L40s GPUs show the approach incurs less than 50ms downtime and under 10% overhead on TTFT and TPOT.
Achieve near-instant (<50ms) service downtime when dynamically reconfiguring LLM inference pipelines across heterogeneous GPUs in serverless environments.
With the rapid advancement of large language models (LLMs), efficiently serving LLM inference under limited GPU resources has become a critical challenge. Recently, an increasing number of studies have explored applying serverless computing paradigms to LLM serving in order to maximize resource utilization. However, LLM inference workloads are highly diverse, and modern GPU clusters are inherently heterogeneous, making it necessary to dynamically adjust deployment configurations online to better adapt to the elastic and dynamic nature of serverless environments. At the same time, enabling such online reconfiguration is particularly challenging due to the stateful nature of LLM inference and the massive size of model parameters. In this paper, we propose a dynamic pipeline reconfiguration approach that enables online adjustment of pipeline configurations while minimizing service downtime and performance degradation. Our method allows the system to select the optimal pipeline configuration in response to changing workloads. Experimental results on heterogeneous GPU platforms, including NVIDIA A100 and L40s, demonstrate that our migration mechanism incurs less than 50 ms of service downtime, while introducing under 10% overhead on both time-to-first-token (TTFT) and time-per-output-token (TPOT).
