Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the degradation of semantic structure in robotic action representations during fine-tuning of Vision-Language-Action (VLA) models, revealing that this erosion negatively impacts task success and generalization. By leveraging insights from mirror neuron theory, the authors propose a novel method to anchor action representations to a semantic manifold, effectively separating shared and private channels while maintaining the integrity of the deployed model. Their approach demonstrates significant improvements, achieving up to +18.7% in real-world in-distribution tasks and +21.5% in out-of-distribution generalization across various VLA backbones.
Action representations can be anchored to a semantic manifold, leading to substantial gains in both in-distribution and out-of-distribution performance for robotic tasks.
Vision-Language-Action (VLA) models inherit rich semantic representations from pretrained Vision-Language Models, yet fine-tuning on limited robot demonstrations degrades this structure and undermines generalization. A fundamental question therefore arises: what constitutes a good action representation? Inspired by the mirror neuron theory's insight that observation and execution share an intention-level encoding, we examine whether a robot's action representations preserve the semantic structure captured by pretrained encoders. Systematic probing confirms that this structure erodes during finetuning, and that its quality synchronizes with both task success and out-of-distribution generalization. We further introduce a plug-and-play method that anchors action representations to a semantic manifold while decomposing representations into a shared semantic channel and a private channel, all discarded at inference, leaving the deployed model unchanged. Validated on different VLA backbones across simulation and real-world benchmarks, our method yields up to +18.7% on real-world in-distribution tasks and +21.5% on out-of-distribution generalization.