Search papers, labs, and topics across Lattice.
This paper introduces a novel framework, In-Context VLA, which enhances Vision-Language-Action models by integrating grounded language comprehension through in-context post-training and an agentic tool-use interface. The authors demonstrate that traditional chain-of-thought reasoning leads to degraded low-level control due to conflicting objectives between reasoning and action, whereas their approach allows models to effectively interpret and utilize language in a grounded manner. Empirical results across various benchmarks and real-world tasks show that this method achieves state-of-the-art performance and efficiency, surpassing conventional CoT-based strategies.
Grounded language comprehension, rather than free-form reasoning, is the key to unlocking superior performance in Vision-Language-Action models.
Vision-Language-Action (VLA) models have become the dominant recipe for generalist manipulation, yet they are almost universally trained by behavior cloning: a policy imitates expert action chunks conditioned on a static image and a fixed instruction. A natural remedy is to inject explicit reasoning through textual chain-of-thought (CoT). We show, both empirically and analytically, that free-form textual CoT degrades low-level control: the reasoning it produces is ungrounded, its latency breaks closed-loop timing, and, crucially, the reasoning and action tokens are optimized against conflicting objectives so that the policy learns to narrate rather than to act. We argue that what a VLA needs is not the ability to generate language, but the ability to consume grounded language. To this end we introduce \textbf{\ourmethod{}}, a framework that endows a VLA with language competence through (i) in-context post-training, in which perceptual evidence is injected as structured context and the model is supervised only on actions, and (ii) an agentic tool-use interface, in which the policy queries open-vocabulary detectors, monocular depth, and a vision--language model to actively acquire task-relevant information. Rather than emitting a single templated caption, our data engine produces diverse, paraphrased, and evidence-conditioned spatial descriptions, so that the policy learns to interpret language it has never seen verbatim. Across the RoboCasa-GR1, SimplerEnv, and LIBERO simulation benchmarks, together with 8 real-world robot manipulation tasks, our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches under matched configurations.
Tsinghua AI