Search papers, labs, and topics across Lattice.
This paper introduces PROSE, a training-free method for egocentric scene registration that leverages pretrained vision-language models to achieve robust cross-scan matching. By transforming RGB sequences into object-level 3D scene graphs and employing object height priors, PROSE effectively addresses the challenges posed by blurry and partially overlapping views typical in egocentric captures. The method demonstrates superior registration accuracy compared to both geometric and learned scene-graph baselines on multiple benchmarks, highlighting its potential for enhancing persistent spatial memory in robotics and AR systems.
Achieving state-of-the-art registration accuracy without any learned parameters or depth sensors, PROSE redefines the landscape of egocentric scene understanding.
Registering two captures of the same indoor space taken at different times underpins persistent spatial memory for robots and AR systems, yet the realistic version of this task is egocentric and its most scalable form is RGB-only. Head-mounted cameras yield blurry, fast-moving, partially overlapping views from which dense geometry is hard to recover. Classical registration leans on exactly the clean point clouds this setting lacks, while learned scene-graph methods require a pre-built or annotated graph and a trained matcher that we find brittle under egocentric data. We take a different route, using a pretrained vision-language model as the source of both scene understanding and cross-scan matching. Our method, PROSE (Prompted Scene rEgistration), lifts each RGB sequence into an object-level 3D scene graph using off-the-shelf foundation models for geometry, segmentation, and language, then prompts the same VLM to match object instances across the two RGB sequences. To make this matching tractable and reliable, we leverage object heights as a prior and verify each proposed match with a paired same/different query, then solve for the rigid transform by hypothesizing a candidate per matched object and selecting the one with the strongest geometric consensus. PROSE adds no learned parameters and requires no depth sensor, training, or annotated graph. On the egocentric Aria Digital Twin and Aria Everyday Activities benchmarks, it outperforms both geometric and learned scene-graph baselines in registration accuracy, on ground-truth and RGB-reconstructed point clouds alike, and the scene graph it produces transfers directly to downstream tasks.
ETH