Search papers, labs, and topics across Lattice.
This paper introduces HeiSD, a hybrid speculative decoding framework tailored for accelerating embodied Vision-Language-Action (VLA) models in robotics. HeiSD strategically combines drafter-based and retrieval-based speculative decoding, leveraging a verify-skip mechanism and sequence-wise relaxed acceptance for retrieval-based drafts, and a kinematic-based fused metric to dynamically determine the optimal hybrid boundary. Experiments show HeiSD achieves up to 2.45x speedup in simulation and 2.41x in real-world scenarios without sacrificing task success.
Robots can think (and act) twice as fast: HeiSD's hybrid speculative decoding turbocharges embodied agents by intelligently switching between draft and retrieval strategies.
Vision-Language-Action (VLA) Models have become the mainstream solution for robot control, but suffer from slow inference speeds. Speculative Decoding (SD) is a promising acceleration method which can be divided into two categories: drafter-based SD and retrieval-based SD. Existing methods fail to analyze the advantages and disadvantages of these two types of SD in VLA models, leading to their sole application or optimization. In this paper, we analyze the trajectory patterns of robots controlled by the VLA model and derive a key insight: the two types of SD should be used in a hybrid manner. However, achieving hybrid SD in VLA models poses several challenges: (1) draft rejection and persistent errors in retrieval-based SD; (2) difficulty in determining the hybrid boundary. To address these, we propose the HeiSD framework. We propose a retrieval-based SD optimization method in HeiSD,which contains a verify-skip mechanism and a sequence-wise relaxed acceptance strategy. Moreover, we proposed a kinematic-based fused metric in HeiSD to automatically determine the hybrid boundary. Experimental results demonstrate that HeiSD attains a speedup of up to 2.45x in simulation benchmarks and 2.06x~2.41x in real-world scenarios, while sustaining a high task success rate.
