Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Vision-TL-Action, a novel approach for generating action trajectories in robotic tasks using multi-view images and a coordinate-free temporal logic (TL) syntax graph. By employing bidirectional cross-attention to fuse TL-node tokens with spatial visual tokens, the model effectively translates visual observations into actionable trajectories without relying on exact object geometry during inference. The approach achieves a Success@1024 rate of 67.45% on the Panda task, surpassing the oracle-state baseline, and demonstrates comparable performance on the AntMaze task, highlighting the importance of semantic grounding in trajectory generation.
Achieving over 67% success in generating action trajectories directly from visual inputs and temporal logic without needing object geometry at inference is a game-changer for robotic task execution.
Temporal logic (TL) provides a compositional language for the formulation of long horizon robotic tasks, but existing TL-conditioned trajectory generators can sidestep perception-to-symbol binding by encoding exact object geometry in the task graph. We introduce \emph{Vision-TL-Action}, which generates action trajectories from multi-view images, a coordinate-free TL syntax graph, and the robot initial state. TL-node tokens and spatial visual tokens are fused through bidirectional cross-attention, and the resulting representation conditions a flow-matching trajectory generator. Visual tokens are augmented only with normalized image-plane locations and camera-view identifiers, while a training-only predicate-to-region objective encourages grounding to referenced objects. Consistent with prior work in this domain, we evaluate the model using Success@$K$, the fraction of tasks for which at least one of K sampled trajectories satisfies the TL specification. On Panda task, our model achieves 67.45% Success@1024, compared with 59.11% for the oracle-state baseline. On AntMaze task, it achieves 96.35% Success@256, comparable to the oracle result of 96.88%. Resolution and intervention studies show that spatial detail depends on semantic grounding and predicate identity affects both attention and performance. These results demonstrate a direct mapping from visual observations and structured TL goals to action trajectories without requiring object geometry at inference. Code is available at https://github.com/AricLau07/vision-tl-action.