Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces 3D-Prog, a framework that enables powerful 2D VLMs to perform 3D tasks by incorporating Canonical Coordinate Framing (CCF) for unified 3D representation and Task-Adaptive Feedback (TAF) for iterative reasoning. CCF addresses challenges like axis ambiguity and scale inconsistency by anchoring inputs and outputs to a shared Euclidean space, while TAF allows 2D VLMs to adaptively refine their understanding and actions within their native visual context. Without retraining, 3D-Prog achieves high-quality, interpretable 3D understanding, manipulation, and generation across diverse object- and scene-level tasks.
Powerful 2D VLMs can be transformed into geometry-aware 3D programmers capable of complex manipulation and generation tasks, simply by grounding their outputs in a canonical coordinate system and employing adaptive feedback loops.
Recent vision-language models (VLMs) exhibit remarkable generalization and reasoning abilities, yet 3D understanding in these models is limited by data scale, training diversity, and reasoning capacity. Instead of naively extending these models into 3D, we take a different approach: we enable powerful 2D VLMs to operate reliably in 3D by introducing 3D grounding and iterative feedback loops with two novel concepts: Canonical Coordinate Framing (CCF) and Task-Adaptive Feedback (TAF). CCF serves as a unified visual representation that anchors both inputs and outputs to a shared Euclidean coordinate system, solving common challenges in 3D grounding such as axis ambiguity, inconsistent metric scale, and floating references. Complementary to this structured framing of the 3D inputs, TAF closes the reasoning loop with task-adaptive dynamic feedback that enables 2D VLMs to perform varied open-vocabulary tasks within their native visual context. Building on this foundation, we introduce 3D-Prog, a 3D understanding, reasoning, and generation framework that jointly employs the capabilities of CCF and TAF together with powerful VLMs. Without requiring any retraining, 3D-Prog performs open-vocabulary 3D understanding, manipulation, and generation across both object-level and scene-level tasks. Our experiments show that the joint use of CCF and TAF transforms 2D VLMs into geometry-aware 3D programmers, achieving consistent, interpretable, and high-quality results across diverse 3D tasks.