Search papers, labs, and topics across Lattice.
This paper presents a novel framework for dense metric depth completion utilizing sparse measurements from Direct Time-of-Flight (dToF) sensors, addressing the limitations posed by their high cost and low-resolution outputs. The proposed method employs a depth-guided dual-branch Vision Transformer encoder, which separately processes RGB images and dToF data, and incorporates a masked joint attention module to effectively integrate depth information. Achieving strong zero-shot generalization across multiple datasets and real dToF devices, the model outperforms existing techniques in both accuracy and computational efficiency, marking a significant advancement in depth perception technologies.
Achieving dense metric depth completion from sparse dToF sensors, our framework outperforms state-of-the-art methods while being trained solely on synthetic data.
Direct Time-of-Flight (dToF) sensors provide highly accurate metric depth and are more robust than indirect ToF systems in challenging real-world conditions. However, their high manufacturing cost and limited photodiode array size produce depth maps that are extremely sparse, low-resolution, and noisy, making them unsuitable for VR/XR, robotics, and 3D perception tasks that require dense metric depth. Existing monocular and depth completion methods struggle to handle the unique sampling patterns and hardware artifacts of dToF devices, and their performance often deteriorates significantly under severe sparsity or noise. We present a generalizable framework for dense metric depth completion from sparse dToF measurements, capable of operating across diverse sensor types, sparsity levels, and noise conditions. Our model employs a depth-guided dual-branch Vision Transformer encoder that processes RGB images and sparse dToF measurements separately, while a masked joint attention module allows depth tokens to reliably guide image features without being overwritten by them. A lightweight decoder reconstructs dense metric depth efficiently, without diffusion-based or refinement-heavy post-processing. To address the scarcity of paired training data, we introduce a comprehensive dToF simulation pipeline that reproduces the characteristics of flash, sub-VGA flash, and rotating sensors, including hardware-induced degradation, irregular sparsity, and realistic noise distributions. Trained entirely on synthetic data, our model achieves strong zero-shot generalization across 6 datasets and 3 real dToF devices, outperforming state-of-the-art approaches in both accuracy and computational efficiency. This establishes a robust and practical solution for dense metric depth completion from sparse direct ToF sensors. Our code and models are open-sourced. See https://vclab.kaist.ac.kr/cvpr2026p3.
