Search papers, labs, and topics across Lattice.
This paper introduces FPCO-Dialog, a benchmark designed to evaluate the correction and cooperation capabilities of vision-language models (VLMs) in multi-turn dialogues where users operate under false premises. By employing a structured protocol with 1,080 images and 10,800 question turns, the authors assess how various VLMs respond to repeated false premises, revealing significant differences in their correction tendencies and dynamics. The findings highlight the need for improved model performance in handling persistent inaccuracies in user assumptions, with implications for the reliability of VLMs in real-world applications.
VLMs exhibit striking variability in their ability to correct repeated false premises, with some models showing persistent inaccuracies that could mislead users.
Vision-language models (VLMs) are increasingly deployed in multi-turn settings where users may describe visual content with incorrect assumptions. Yet existing evaluations rarely isolate how models respond when the same visually grounded false premise persists across dialogue turns. We introduce FPCO-Dialog, a benchmark for evaluating correction and cooperation behavior in VLMs under repeated false premises. FPCO-Dialog contains 1,080 images and 10,800 question turns, stratified by visual complexity, object category, and false-premise class, and uses a 10-turn protocol in which a correct dialogue prefix is followed by repeated false-premise referring expressions. We evaluate 20 commercial and open-source VLMs with a model-agnostic protocol and CorrTP@K, a correction-rate metric over false-premise turns, scored by two independent detectors. FPCO-Dialog reveals substantial and persistent cross-model differences in aggregate correction tendency, model-specific turn-wise dynamics, and systematic variation across false-premise types under the benchmark's substitution distribution. The dataset, evaluation protocol, model outputs, detector labels, and code are available.
