Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a controlled evaluation framework for assessing interactive visual grounding in large vision-language models (LVLMs), emphasizing the need for dialogue in real-world referencing scenarios. The study reveals that LVLMs significantly underperform compared to human baselines, particularly when initial target descriptions are absent and information must be acquired through interaction. Additionally, the models exhibit poor calibration, often overestimating their confidence relative to their actual accuracy, highlighting critical shortcomings in their interactive capabilities.
LVLMs struggle with interactive visual grounding, often failing to effectively acquire target information through dialogue, especially when starting from scratch.
Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a central property of real-world reference: target information is often incomplete, ambiguous, and established through interaction. We introduce a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), varying how much target information is provided upfront and how much must be acquired through dialogue. Across four human-grounded visual contexts and four interaction protocols, current LVLMs perform significantly below task-level human baselines. Interaction can help when follow-up questions refine or repair an initial target description. Performance is lowest when no initial description is provided and target information must be acquired through questions, indicating that proactive question-driven grounding remains difficult. LVLMs are also poorly calibrated, often reporting confidence that exceeds their empirical accuracy. Follow-up studies confirm these patterns across varied description sources (human versus AI), reasoning efforts, repeated interactions, description providers, and visual contexts. Overall, interactive visual grounding remains an important challenge, requiring visual matching, information seeking and synthesis.