Search papers, labs, and topics across Lattice.
This paper introduces TrAct, a robot decision-making framework that leverages visual tracks to enhance the alignment between robot control and visual prediction. By integrating a Vision-Language-Action-and-Track model, a track-conditioned world model, and a vision-language reward model, TrAct significantly improves task success rates in both simulation (from 27% to 55%) and real-world scenarios (from 49% to 76%) compared to existing methods. The findings highlight that visual tracks serve as a powerful embodiment-agnostic interface, facilitating better world modeling and generalization in robotic systems.
Visual tracks can double a robot's task success rate by bridging the gap between control and visual prediction.
Robot actions are inherently embodiment-specific and only weakly aligned with image-space visual changes, limiting their effectiveness as conditioning signals for robot world models. In contrast, visual tracks provide an embodiment-agnostic representation of how task-relevant points move through a scene, offering dense image-space guidance for accurate and spatially precise future video prediction. Building on this observation, we propose TrAct, a world-model-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction. TrAct consists of three components: a Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction; a track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks; and a vision-language reward model (VLAC) that scores the predicted outcomes. At inference time, VLAT generates candidate action-track pairs, TWM rolls out their visual consequences, and VLAC selects the track whose predicted outcome best satisfies the instruction; the action paired with the selected track is then executed by the robot. Experiments on the proposed LIBERO-INTEGRAL benchmark and real-world Franka manipulation show that TrAct improves success rates from 27% to 55% in simulation and from 49% to 76% on real-world tasks compared with the strong VLA baseline $\pi_{0.5}$. Furthermore, TWM consistently improves video prediction quality over the action-conditioned world model (AWM). These results demonstrate that visual tracks provide an effective shared interface between robot control and visual prediction, enabling more accurate world modeling and stronger robot generalization.
