Search papers, labs, and topics across Lattice.
This paper introduces SALT, a Semantically ALigned action Tokenizer, which enhances vision-language-action models by incorporating verb-grounding information into action representations. By addressing the limitations of traditional reconstruction-based tokenization methods, SALT enables the recovery of episode instructions from quantized action latents, leading to significant improvements in policy success rates. The results demonstrate that integrating linguistic structure into action representations can enhance language-conditioned control in robotic systems, achieving an average success rate of 71.9% in SimplerEnv compared to lower rates for existing methods.
Action representations that align with language can boost robotic control success rates by over 29% compared to traditional methods.
Action verbs describe not only the physical outcomes of actions, but also how those actions are performed. Yet action representations in vision-language-action models (VLAs) are typically optimized for reconstruction under L1/L2 losses in raw action space, where numerical proximity need not reflect linguistically meaningful distinctions. On BridgeV2, we show that action trajectories contain verb-grounding information beyond visual state changes, and that reconstruction-only discrete tokenization systematically erodes this information. To address this problem, we introduce SALT, a Semantically ALigned action Tokenizer that augments a VQ-VAE-style tokenizer with an auxiliary objective requiring a frozen vision-language model to recover the episode instruction from quantized action latents. Policies trained with SALT achieve 71.9% average success in SimplerEnv, compared with 42.7% for a reconstruction-only VQ-VAE tokenizer and 31.2% for FAST. SALT also develops verb-specialized codes while maintaining reconstruction fidelity. These results show that robot action trajectories provide a source of language grounding and that preserving this structure in action representations can substantially improve language-conditioned control.
