Search papers, labs, and topics across Lattice.
WA-SpecDec introduces a world-aware speculative decoding framework for vision-language-action (VLA) models that enhances the efficiency of robot control generation by incorporating physical scene awareness during the prefill stage. By allowing for a more nuanced acceptance of action-token deviations based on the context of the environment, the method significantly improves task success rates and reduces near-contact failures. The results show a 1.5x speedup in matched-success rates compared to traditional VLA speculative decoding, while also decreasing near-contact failure rates by 18.6%.
Injecting world-awareness into speculative decoding leads to a 1.5x speedup in task success rates while reducing near-contact failures by nearly 19%.
Vision-language-action (VLA) policies generate robot controls autoregressively, making closed-loop latency dominated by repeated target-model forward passes. Speculative decoding reduces this cost by verifying blocks of draft action tokens in parallel, and recent VLA methods further relax token-level acceptance because small differences in action-token space often map to similar continuous controls. However, this relaxation remains scene-agnostic. A fixed token-distance tolerance treats the same action-token deviation as equally safe across states, although deviations that are harmless in free space can cause collisions or grasp failures near contact. We propose WA-SpecDec, a world-aware speculative decoding framework that injects world-model-derived physical scene awareness during the VLA prefill stage, producing shared world-aware prefill states for draft proposal and target verification without changing the relaxed acceptance rule. Across three state-of-the-art relaxed acceptance schemes, WA-SpecDec preserves higher task success under looser relaxation and enables longer accepted prefixes. At comparable-success operating points, WA-SpecDec achieves a 1.5x matched-success speedup over VLA speculative decoding alone and reduces near-contact failure (NCF) by 18.6% on average relative to the corresponding speculative baselines.
