Search papers, labs, and topics across Lattice.
This paper critiques the conventional approach of using generated robot videos for training Vision-Language-Action (VLA) models, arguing that the geometric content captured in these videos should guide visual perception rather than attempting to recover control signals from synthesized pixels. The authors introduce the Geometry-guided Representation Alignment (GRA) method, which leverages pose estimation and calibrated projection to extract geometric waypoints from human videos, while training the action head solely on real demonstrations. Experimental results show that GRA significantly outperforms pseudo-action baselines and narrows the performance gap compared to models trained with a larger number of real demonstrations, highlighting the effectiveness of preserving geometric information in VLA training.
GRA reveals that grounding VLA models in geometric representations from generated videos can outperform traditional methods that attempt to extract control signals from the same data.
Vision-Language-Action (VLA) models require large-scale video-action pairs, yet real teleoperation remains scarce. While generated robot videos offer a scalable alternative, existing methods treat them as real robot data by recovering pseudo-actions from synthesized pixels. We argue that deriving low-level control from generated visuals is a mismatched abstraction. A video captures only \emph{geometry}: the spatial trajectory representing the \emph{where} of a task. A real demonstration captures \emph{control}: the exact motor commands representing the \emph{how}. Human-to-robot video generation preserves these unequally: the visible geometry survives the generation process, while the underlying control signals are lost. This \textbf{Asymmetric Preservation Principle} dictates a clean rule: this surviving geometry should solely supervise visual perception, leaving control to real demonstrations. Following this principle, we propose \textbf{GRA} (\textbf{G}eometry-guided \textbf{R}epresentation \textbf{A}lignment), which extracts the geometric content as future 2D end-effector waypoints, computed from the source human video through pose estimation, retargeting, simulation, and calibrated projection, and routes them to the VLA vision backbone via an auxiliary 2D head. The action head is trained on real demonstrations only. During fine-tuning, the waypoint loss persists as a \textbf{spatial representation anchor} that prevents the backbone from losing its geometric grounding. On real-robot tasks, GRA outperforms pseudo-action baselines under matched data budgets and narrows the gap to policies trained with substantially more real demonstrations, suggesting that correctly routed geometry bridges generated videos to robot policies more reliably than recovered actions.
NUS