Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Pumpire introduces a unified benchmark for evaluating the metric point-pair distance estimation capabilities of 3D foundation models, addressing the limitations of prior methods that assessed depth or geometric similarity separately. The benchmark comprises a large-scale dataset, pumpire-6k, with 100 real-world scenes annotated with ground-truth point-pair distances, enabling direct assessment of point-to-point distance errors. Extensive experiments on 29 baseline configurations reveal the current state of metric scale perception in reconstructed 3D spaces, highlighting a critical but previously overlooked evaluation aspect.
Most 3D foundation models struggle to accurately estimate absolute metric distances, a fundamental capability overlooked by existing benchmarks.
We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors. In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models'point-to-point distance estimation capability, Pumpire directly assesses point-to-point distances from the reconstructed geometry. To this end, we collect a large-scale and diverse dataset (pumpire-6k) comprising 100 real-world scenes, each annotated with physically measured point-pair distances and containing 64 frames, for a total of 6,400 frames. Building on this dataset, we establish a holistic evaluation protocol that covers both image- and video-level 3D foundation models and enables direct assessment of point-pair distance errors and cross-setting comparison. We conduct extensive experiments across 29 baseline configurations of representative 3D foundation models and provide a comprehensive analysis of the results. By offering this benchmark, we target the more fundamental ability to perceive and estimate physical scale in the reconstructed 3D space, which prior evaluation protocols have largely overlooked. The project page can be found at https://pumpire.github.io/