Search papers, labs, and topics across Lattice.
This paper introduces CapDepth, a novel framework for monocular depth estimation (MDE) that utilizes detailed long captions to improve robustness against visual ambiguities in challenging scenarios. By designing a comprehensive caption input template and employing a dynamic caption encoder with progressive masked attention, CapDepth effectively enhances depth-relevant feature extraction. Experimental results demonstrate that CapDepth achieves significant depth error reductions of 25.0% on non-Lambertian surfaces and 22.0% in adverse weather conditions, outperforming existing state-of-the-art methods.
Detailed long captions can reduce depth estimation errors by up to 25% in challenging visual conditions, transforming how we approach monocular depth estimation.
Monocular depth estimation (MDE) faces challenges with non-Lambertian surfaces and adverse weather conditions due to the visual ambiguities inherent in single-image limited information. Existing works address them in isolation via image inpainting or augmentation, yielding limited robustness gains. Language, as a powerful complementary modality to vision, is demonstrated to enhance the visual perception capabilities of vision-language models (VLMs) via detailed long captions. However, prior language-integrated MDE methods fail to fully harness this potential due to short text input with limited information, coarse global text feature learning, and limited language guidance during depth decoding. To address these limitations, we propose CapDepth, a novel framework for robust MDE that leverages guidance from detailed long captions to alleviate visual ambiguities in both challenging scenarios. First, we design a detailed long caption input template that explicitly conveys rich spatial relationships among multiple atom sentences. Second, a dynamic caption encoder is introduced to extract fine-grained depth-relevant text features via progressive masked attention. Finally, we propose a text-adaptive decoder that guides enhanced depth decoding with text features via stable adaptive layer normalization. Extensive experiments validate the efficacy of CapDepth, which outperforms state-of-the-art methods, achieving depth error reductions of 25.0% on non-Lambertian surfaces and 22.0% under adverse weather conditions.
