Search papers, labs, and topics across Lattice.
This paper introduces PIVOT, a comprehensive multi-trajectory dataset and evaluation framework designed to assess the performance of 3D reconstruction methods like NeRFs and 3D Gaussian Splatting under realistic conditions. By incorporating diverse camera trajectories and comparing measured versus optimized poses and intrinsics, the study reveals significant performance discrepancies when transitioning from training to unseen trajectories. Key results indicate that existing benchmarks may overestimate model capabilities, highlighting the importance of realistic evaluation conditions in 3D reconstruction tasks.
Performance gaps in 3D reconstruction methods can exceed expectations by over 40% when evaluated under realistic conditions, challenging the validity of current benchmarks.
Neural radiance fields (NeRFs), 3D Gaussian Splatting (3DGS), and related novel-view synthesis methods are commonly evaluated under capture and reconstruction conditions cleaner than those encountered by robots, drones, and autonomous systems. Benchmarks often rely on reconstruction-friendly trajectories, optimized camera poses and intrinsics, and held-out views sampled from trajectories represented during training. These assumptions can obscure performance with measured poses, reusable camera calibration, and structurally different camera paths. We introduce PIVOT (Pose, Intrinsics and Viewpoint Oriented Testbed), a multi-trajectory dataset, processing pipeline, and evaluation framework for independently studying these factors. PIVOT captures each scene using diverse camera trajectories and retains, where available, both sensor-derived measured poses and COLMAP-optimized poses, together with calibrated and optimized camera intrinsics. It defines three benchmark families: (1) seen versus unseen trajectory novel-view generalization, (2) measured versus optimized pose sensitivity, and (3) calibrated versus optimized intrinsics sensitivity. We also introduce a directed pose-space Chamfer distance to quantify how well training poses cover an evaluation trajectory. PIVOT v1 contains five real-world scenes captured with a DJI Mini 4 Pro and provides an open processing and Nerfstudio-based evaluation toolchain. Benchmark results show a consistent quality gap between held-out views on represented trajectories and unseen trajectories, as well as substantial sensitivity to pose source and camera intrinsics.
