Search papers, labs, and topics across Lattice.
This paper introduces DreamTraj, a novel method for predicting 6-DoF object trajectories using only a single RGB image and a natural language instruction, bypassing the need for video, depth, or CAD models during inference. The authors address the limitations of existing datasets and predictors by creating the MOVE dataset, which includes 5,038 object-centric trajectories paired with fine-grained language annotations. DreamTraj achieves state-of-the-art performance in both translation and rotation while operating 4.6 times faster than traditional generate-then-extract methods, marking a significant advancement in trajectory prediction from unrendered video diffusion latents.
DreamTraj sets a new benchmark in 6-DoF trajectory prediction by leveraging language and a single image, outperforming traditional methods that rely on privileged inputs.
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.
