Search papers, labs, and topics across Lattice.
GATOR addresses the challenge of reconstructing complete, scene-aligned 3D objects from sparse, casual images by employing a generative and agentic framework that integrates local feature mixing with text-guided semantic conditioning. This approach first recovers textured object assets and their scene-relative pose by coupling patch-aligned RGB, mask, and pointmap features, then refines them through an observation-guided edit-render-review loop driven by a multimodal agent. GATOR demonstrates strong geometric and appearance fidelity across diverse datasets, achieving efficient reconstruction and simulation readiness.
Casual images are now enough to generate high-fidelity, scene-aligned 3D objects with accurate pose, even from sparse observations.
Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness. Project page: https://research.nvidia.com/labs/lpr/gator/
