Search papers, labs, and topics across Lattice.
This paper introduces SpecKV, an adaptive controller for speculative decoding that dynamically selects the speculation length (γ) based on draft model signals. They profiled speculative decoding across various tasks, speculation lengths, and compression levels, demonstrating that the optimal γ varies significantly with compression and that draft model confidence and entropy are strong predictors of acceptance rate. SpecKV uses a small MLP trained on these signals to maximize expected tokens per speculation step, achieving a 56% improvement over a fixed γ=4 baseline.
Pushing speculative decoding to new heights, SpecKV adaptively tunes speculation length based on draft model confidence, achieving a 56% speedup compared to fixed-length speculation, especially crucial for compressed models.
Speculative decoding accelerates large language model (LLM) inference by using a small draft model to propose candidate tokens that a larger target model verifies. A critical hyperparameter in this process is the speculation length~$γ$, which determines how many tokens the draft model proposes per step. Nearly all existing systems use a fixed~$γ$ (typically~4), yet empirical evidence suggests that the optimal value varies across task types and, crucially, depends on the compression level applied to the target model. In this paper, we present \textbf{SpecKV}, a lightweight adaptive controller that selects~$γ$ per speculation step using signals extracted from the draft model itself. We profile speculative decoding across 4~task categories, 4~speculation lengths, and 3~compression levels (FP16, INT8, NF4), collecting 5,112 step-level records with per-step acceptance rates, draft entropy, and draft confidence. We demonstrate that the optimal~$γ$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate (correlation~$\approx 0.56$). SpecKV uses a small MLP trained on these signals to maximize expected tokens per speculation step, achieving a 56.0\% improvement over the fixed-$γ$=4 baseline with only 0.34\,ms overhead per decision ($<$0.5\% of step time). The improvement is statistically significant ($p < 0.001$, paired bootstrap test). We release all profiling data, trained models, and notebooks as open-source artifacts.
