Search papers, labs, and topics across Lattice.
The paper introduces MM-AQA, a new benchmark for evaluating abstention capabilities in multimodal reasoning systems by creating unanswerable instances from answerable ones through transformations along visual modality dependency and evidence sufficiency. Experiments on VLMs and multi-agent systems reveal that VLMs rarely abstain under standard prompting, and while MAS improves abstention, it introduces an accuracy-abstention trade-off. The findings suggest that abstention-aware training is crucial for effective multimodal abstention, as current models struggle with degraded or contradictory evidence.
VLMs confidently hallucinate answers even when presented with insufficient or contradictory multimodal evidence, highlighting a critical gap in their reliability.
Effective abstention (EA), recognizing evidence insufficiency and refraining from answering, is critical for reliable multimodal systems. Yet existing evaluation paradigms for vision-language models (VLMs) and multi-agent systems (MAS) assume answerability, pushing models to always respond. Abstention has been studied in text-only settings but remains underexplored multimodally; current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes. We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes: visual modality dependency and evidence sufficiency. Evaluating three frontier VLMs spanning closed and open-source models and two MAS architectures across 2079 samples, we find: (1) under standard prompting, VLMs rarely abstain; even simple confidence baselines outperform this setup, (2) MAS improves abstention but introduces an accuracy-abstention trade-off, (3) sequential designs match or exceed iterative variants, suggesting the bottleneck is miscalibration rather than reasoning depth, and (4) models abstain when image or text evidence is absent, but attempt reconciliation with degraded or contradictory evidence. Effective multimodal abstention requires abstention-aware training rather than better prompting or more agents.
