Search papers, labs, and topics across Lattice.
This paper introduces SafeSpec, a novel framework that enhances safety in speculative inference for large language models (LLMs) by integrating risk estimation into the verification process. By employing a lightweight latent safety head, SafeSpec allows for the simultaneous evaluation of semantic validity and safety, enabling the model to recover safe continuations through rollback and reflective multi-sampling when unsafe generations are detected. The results show that SafeSpec significantly improves the safety-efficiency trade-off, achieving a 15% reduction in attack success rates on Qwen3-32B while also providing a 2.06x speedup in inference for benign workloads.
SafeSpec achieves a remarkable 15% reduction in attack success rates while accelerating inference by over 2x, effectively marrying safety and speed in LLMs.
Speculative inference accelerates large language model (LLM) decoding but provides no inherent safety guarantees. Existing safety defenses are largely incompatible with speculative inference: they either introduce additional computation or disrupt the draft-verify mechanism, negating acceleration benefits. This reveals a fundamental incompatibility between current safety methods and speculative decoding. We propose SafeSpec, a safety-aware speculative inference framework that integrates risk estimation directly into the verification process. SafeSpec attaches a lightweight latent safety head to the target model to jointly evaluate semantic validity and safety in a single forward pass. When unsafe generations are detected, SafeSpec applies rollback and safety-guided reflective multi-sampling to recover safe continuations rather than terminating generation. We model jailbreak attacks as distributional shifts over generative trajectories, where adversarial prompts increase the probability of harmful continuations without eliminating safe ones. Under this model, SafeSpec performs risk-aware trajectory recovery within the speculative decoding process. Across multiple models and adversarial benchmarks, SafeSpec achieves a substantially improved safety-efficiency trade-off. On Qwen3-32B, SafeSpec reduces attack success rates by 15% while preserving a 2.06x inference speedup on benign workloads, demonstrating that speculative acceleration and inference-time safety can be jointly optimized.
