Search papers, labs, and topics across Lattice.
This paper introduces AgentSpec, a novel speculative decoding algorithm designed to enhance the inference efficiency of large language model (LLM) agents while maintaining generation quality. By systematically analyzing existing speculative decoding methods, the authors identify key factors contributing to speed degradation, such as high rejection rates and under-utilization of token budgets. AgentSpec mitigates these issues through structure-isolated drafting and redundancy-aware budget allocation, resulting in significantly improved performance across multiple workloads and LLM families.
AgentSpec slashes response times for LLM agents by addressing high rejection rates and optimizing token budgets, outperforming existing methods.
Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.
