Search papers, labs, and topics across Lattice.
This paper introduces Track4Action, a framework that leverages a world-centric 3D tracker to enhance vision-language-action (VLA) policies by distilling crucial geometric and motion information from aligned demonstration clips. By encoding the transitions of actions into a pooled tracker feature, the framework enables learnable track queries to infer and condition action decisions based on current observations. The results demonstrate significant performance improvements across various benchmarks, achieving an 82.3% success rate on zero-shot LIBERO-Plus and outperforming existing alignment-free methods by substantial margins.
Action-aligned 3D tracker features can boost VLA policy performance by over 25 points, revolutionizing how robots learn from demonstrations.
Action labels tell a vision-language-action (VLA) policy which robot commands to imitate, but not how those commands change the 3D world. The aligned demonstration clip contains this missing supervision because its $K$ frame transitions record the geometry, motion, visibility, and camera change produced during the corresponding $K$ actions. We introduce Track4Action, a framework that distills this realized transition from a frozen world-centric 3D tracker into a current-observation VLA policy. During training, Track4World encodes the clip $V_{t:t+K}$ into a pooled tracker feature. Learnable track queries infer this feature from current VLA hidden states, match it in a shared space, and condition a flow-matching action head through a feature-wise gate. The tracker feature only defines the alignment target, so neither the clip nor the tracker is used at deployment. Track4Action reaches 82.3% on zero-shot LIBERO-Plus, improving the alignment-free variant by 7.6 points and LaMP by 3.0 points. It obtains 80.44% and 81.48% on the clean and randomized RoboTwin 2.0 splits, and 67.5% average success across four physical bimanual tasks, 25.0 points above the alignment-free variant. The gains across simulation and physical tasks support action-aligned 3D tracker features as privileged supervision for tracker-free VLA deployment. Our project page is available at https://wing0night.github.io/track4action-project-page.
