Search papers, labs, and topics across Lattice.
This paper introduces AsymSpec, an innovative asymmetric speculative decoding framework designed to enhance the efficiency of agentic LLMs by allowing a lightweight drafter to utilize full context while a larger verifier operates on a compressed version. By implementing a contrastive $\delta$-fusion of logits and a divergence-aware acceptance gate, AsymSpec achieves nearly 90% of full-context accuracy while significantly improving throughput by 1.3 to 1.7 times at a fraction of the compute cost. The findings underscore the advantages of asymmetric context access, particularly in scenarios where compression risks losing essential reasoning signals.
AsymSpec achieves 90% accuracy with 1.7x speedups by leveraging asymmetric context access, redefining efficiency in agentic LLMs.
Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive $\delta$-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches $\approx 90\%$ of full-context accuracy on average, delivering $1.3$--$1.7\times$ throughput speedups at $0.2$--$0.3\times$ the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.
