Search papers, labs, and topics across Lattice.
The paper introduces SAFE-Pruner, a token pruning framework for accelerating vision-language-action (VLA) models by incorporating attention cues from future layers into pruning decisions. It leverages the observation of semantic attention consistency across execution steps to forecast token saliency in deeper layers, preventing premature removal of critical tokens. Experiments on simulated and real-world robotic control tasks show that SAFE-Pruner achieves up to 1.89x speedup with minimal performance degradation, outperforming existing methods.
Don't let shallow-layer cues fool you: SAFE-Pruner uses "future sight" to prune vision-language-action models, achieving 1.89x speedup without sacrificing task success.
Real-time inference of vision-language-action (VLA) models is essential for robotic control. While visual token pruning has shown strong potential for accelerating inference, most existing methods mainly base pruning decisions on shallow-layer cues and risk discarding visual information required by deep layers. To address this issue, we propose SAFE-Pruner, a plug-and-play pruning framework that incorporates attention cues of future layers into pruning decisions. Specifically, we identify semantic attention consistency, the tendency that VLA models concentrate their attention probability mass on the same semantic entity across execution steps. Based on this observation, we design a forward-looking strategy to forecast the token saliency in deep layers, which prevents the premature removal of critical tokens and leads to more stable acceleration. We further introduce an adaptive subtask division strategy to detect abrupt attention shifts, thereby improving forecasting accuracy and pruning reliability. Extensive experiments in simulation and real-world settings demonstrate that our method achieves up to 1.89x speedup with a minimal degradation in success rate of less than 1.7%, while outperforming state-of-the-art methods by up to 1.9%.
