Search papers, labs, and topics across Lattice.
This paper introduces Depth-Ordinal Prompting (DOP), a novel approach that transforms monocular depth information into text-based ordinal cues to enhance spatial reasoning in vision-language models (VLMs). The authors demonstrate that while depth maps can hinder performance due to their noisy nature, conveying depth as text significantly improves the model's ability to reason about object relationships in 3D space. Their experiments reveal that DOP consistently outperforms existing training-free methods and maintains effectiveness even in scenarios where original images are strong, highlighting the importance of how depth information is presented to the model.
Conveying depth as text rather than images can significantly boost spatial reasoning in vision-language models, challenging conventional approaches.
Vision-language models (VLMs) are expected to reason about physical space -- which object is closer, what lies behind what, and how objects are arranged in 3D -- yet they still struggle with such spatial judgments. A natural remedy is to show the model a depth map, but we find that this can make performance worse. We show that depth is not absent: it reaches the language model, but becomes difficult to access for downstream reasoning, while rendered pseudo-depth maps act as noisy auxiliary images that frozen VLMs cannot easily regulate. We propose Depth-Ordinal Prompting (DOP), a training-free method that converts monocular depth into a single question-targeted ordinal text cue at the queried objects, without adding a depth image, training a module, injecting features, or using labels. Our key finding is form dependence: the same depth signal can hurt when shown as an image but help when told as text.Across benchmarks, models, and depth estimators, DOP improves spatial reasoning when pseudo-depth provides reliable object-level ordering and remains largely neutral in strong original-image regimes. It is also competitive with the strongest training-free depth-prompting alternative while being simpler and more targeted.
NUS