Search papers, labs, and topics across Lattice.
SpecGuard is introduced, a verification-aware speculative decoding framework that uses model-internal signals for step-level verification in multi-step reasoning tasks. It samples multiple draft candidates at each step and selects the most consistent one based on attention-based grounding and log-probability scores. Experiments on reasoning benchmarks demonstrate that SpecGuard improves accuracy by 3.6% and reduces latency by ~11% compared to standard and reward-guided speculative decoding.
Internal model signals alone can guide speculative decoding to achieve both higher accuracy and lower latency in multi-step reasoning, outperforming reward-guided approaches.
Speculative decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose outputs that a stronger target model verifies. However, its token-centric nature allows erroneous steps to propagate. Prior approaches mitigate this using external reward models, but incur additional latency, computational overhead, and limit generalizability. We propose SpecGuard, a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. At each step, SpecGuard samples multiple draft candidates and selects the most consistent step, which is then validated using an ensemble of two lightweight model-internal signals: (i) an attention-based grounding score that measures attribution to the input and previously accepted steps, and (ii) a log-probability-based score that captures token-level confidence. These signals jointly determine whether a step is accepted or recomputed using the target, allocating compute selectively. Experiments across a range of reasoning benchmarks show that SpecGuard improves accuracy by 3.6% while reducing latency by ~11%, outperforming both SD and reward-guided SD.
