Search papers, labs, and topics across Lattice.
This paper systematically compares different strategies for supervising Vision-Language-Action (VLA) models using latent actions, focusing on image-based and action-based approaches. They instantiate and evaluate four integration strategies within a unified VLA baseline, revealing that image-based latent actions improve long-horizon reasoning while action-based latent actions enhance motor coordination. The study finds that directly supervising the VLM with discrete latent action tokens yields the best overall performance, particularly in mixed-data scenarios.
Image-based latent actions are your secret weapon for long-horizon reasoning in VLAs, while action-based latent actions unlock complex motor coordination.
Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens.
