Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
Agentic AI workloads, characterized by long and evolving contexts, degrade speculative decoding performance and increase KV cache overhead. DLite addresses this by employing a history-free approach that reuses target model hidden states for local speculative prediction, bypassing explicit historical context maintenance. This framework achieves significantly higher acceptance lengths and outperforms prior state-of-the-art methods on agentic tasks.
Agentic AI inference can be accelerated by 2-6x with history-free speculative decoding that eliminates KV cache overhead and boosts acceptance rates by up to 70%.
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.