Search papers, labs, and topics across Lattice.
X-Lens introduces a compact feed-forward model for real-time metric depth estimation that effectively integrates data from both fisheye and pinhole cameras. By utilizing learnable calibration tokens and a Jacobian-parameterized distortion bias, the model achieves robust cross-camera consistency and generalization while maintaining high processing speeds of up to 41 FPS with minimal parameters. Extensive evaluations reveal that X-Lens outperforms existing methods, achieving a 25.4% reduction in absolute relative error on the OmniScene-Full dataset while using 88.9% fewer parameters than the strongest baseline.
Achieving superior depth estimation accuracy with 88.9% fewer parameters, X-Lens redefines efficiency in heterogeneous camera systems.
We present X-lens, a compact feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. To support real-time downstream perception, X-lens is built around a geometry-aware heterogeneous camera formulation with two key components. Learnable calibration tokens provide a coarse alignment between fisheye and pinhole projective spaces, while a Jacobian-parameterized distortion bias injected into cross-attention models local projection changes and promotes cross-camera consistency, enabling robust generalization with only 0.04B parameters and up to 41 FPS. The model predicts dense depth together with a global metric scale, avoiding auxiliary reconstruction targets that increase computation and optimization complexity. To learn such cross-camera generalization at scale and depth, X-lens is trained on multiple public datasets and OmniScene, our newly released large-scale synthetic dataset containing approximately 266K synchronized six-view frames, 1.7M individual images, and 103 indoor and outdoor scenes. Extensive experiments on both real-world and synthetic indoor and outdoor datasets demonstrate superior heterogeneous-camera metric depth accuracy, reducing AbsRel by 25.4\% on OmniScene-Full over the strongest baseline while using 88.9\% fewer parameters, with competitive performance on conventional fisheye-only and pinhole-only settings.
