Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces ST-$\pi$, a structured vision-language-action (VLA) model designed to improve robotic manipulation by explicitly modeling spatiotemporal reasoning. The model uses a spatiotemporal VLM to generate chunk-level action prompts with sub-tasks, spatial grounding, and temporal grounding, which then guide a spatiotemporal action expert to predict step-level action parameters. Experiments on a new real-world robotic dataset with structured spatiotemporal annotations demonstrate the effectiveness of the proposed approach.
Robots can now plan manipulation tasks more effectively by explicitly reasoning about space and time, thanks to a new architecture that decomposes actions into causally-ordered chunks.
Vision-language-action (VLA) models have achieved great success on general robotic tasks, but still face challenges in fine-grained spatiotemporal manipulation. Typically, existing methods mainly embed spatiotemporal knowledge into visual and action representations, and directly perform a cross-modal mapping for step-level action prediction. However, such spatiotemporal reasoning remains largely implicit, making it difficult to handle multiple sequential behaviors with explicit spatiotemporal boundaries. In this work, we propose ST-$\pi$, a structured spatiotemporal VLA model for robotic manipulation. Our model is guided by two key designs: 1) Spatiotemporal VLM. We encode 4D observations and task instructions into latent spaces, and feed them into the LLM to generate a sequence of causally ordered chunk-level action prompts consisting of sub-tasks, spatial grounding and temporal grounding. 2) Spatiotemporal action expert. Conditioned on chunk-level action prompts, we design a structured dual-generator guidance to jointly model spatial dependencies and temporal causality, thus predicting step-level action parameters. Within this structured framework, the VLM explicitly plans global spatiotemporal behavior, and the action expert further refines local spatiotemporal control. In addition, we propose a real-world robotic dataset with structured spatiotemporal annotations for fine-tuning. Extensive experiments have been conducted to demonstrate the effectiveness of our model. Our code link: https://github.com/chuanhaoma/ST-pi.