Search papers, labs, and topics across Lattice.
This study empirically investigates the adoption of LLM serving frameworks and methods in open-source software systems, focusing on five specific frameworks: vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. The research reveals that vLLM is the most popular framework, while techniques such as parallel computation, memory management, and network pruning dominate the serving methods employed. Notably, the findings indicate limited multi-framework usage, suggesting that developers tend to rely on a single framework despite the potential benefits of combining complementary capabilities across the serving stack.
vLLM emerges as the leading framework for LLM serving, but developers are missing out on the advantages of multi-framework integration.
Large Language Models (LLMs) are integrated into software systems and AI services, making efficient LLM serving a concern for software engineering. Serving LLMs is challenging because inference requires computation, memory, GPU resources, and execution while maintaining latency and throughput. Although prior research has proposed LLM inference, optimization, and serving techniques and frameworks, little is known about how they are adopted in practice. In this study, we investigate the use of LLM serving frameworks and serving methods in open-source software systems. We identify and analyze five LLM-specific frameworks: vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. We examine how these frameworks and techniques are adopted individually and in combination, how adoption varies across categories of LLMs, and how repositories differ in intent, focus, use case, and architectural design. Our results show that vLLM is the most visible framework in popularity and adoption, while parallel computation, memory management, and network pruning are the most frequently used serving-method categories. Multi-framework usage is limited, suggesting that developers rely on a single serving framework; however, combined frameworks connect complementary capabilities across the serving stack. Framework adoption varies across model families, modalities, model sizes, domain specializations, and deployment settings. Repository-level analysis shows that LLM serving frameworks support applications and architectures, including Reinforcement Learning (RL)-based reasoning, multimodal generation and understanding, microservices, and cloud infrastructure. Overall, this study provides a large-scale empirical characterization of LLM serving framework adoption in practice and offers insights for researchers, framework maintainers, and practitioners working on LLM systems.
