Search papers, labs, and topics across Lattice.
3DWay reformulates robotic trajectory generation by predicting multi-view consistent 2D waypoints from multi-camera observations and resolving them into explicit 3D space via geometric triangulation. This formulation eliminates the free-space 3D depth ambiguities common in standard 2D trajectory representations while directly leveraging the general-purpose visual and semantic priors of pretrained vision-language models. The resulting 3D trajectories reliably improve spatial grounding and manipulation generalization, whether used as guidance for downstream vision-language-action (VLA) models or executed directly.
Standard 2D vision-language models can reliably plan precise 3D robot trajectories simply by enforcing multi-view 2D waypoint consistency and applying classic geometric triangulation.
Intermediate representations are key to bridging the modality gap between generalizable manipulation policies and large-scale pretrained vision-language models (VLMs). Among these, trajectory-based representations compactly represent motion-relevant cues, yet most existing approaches predict trajectories in 2D image space, resulting in intrinsic 3D ambiguity. Moreover, using 2D trajectories with depth still leaves the free-space waypoints ambiguous, limiting reliable 3D reasoning. To address this, we propose predicting 3D consistent waypoints (3DWay) from multi-view images. By reformulating 3D waypoints prediction as generating multi-view consistent 2D waypoints followed by geometric triangulation, we enable explicit 3D motion specification while preserving the strong priors of pretrained VLMs. The predicted waypoints can guide existing VLA models for better generalization or be directly executed on simple tasks. Extensive experiments show that 3DWay substantially improves 3D spatial grounding and vision-language reasoning, demonstrating strong potential for generalizable robot manipulation. Codes will be released at https://github.com/ziqin-h/3DWay.
