Search papers, labs, and topics across Lattice.
This survey organizes the literature on uncertainty-aware multimodal large language models (MLLMs) through a decision-centered framework, highlighting various sources of uncertainty and their impact on model performance. It emphasizes that uncertainty should be assessed not just as a confidence metric but in terms of its ability to enhance decision-making under challenging conditions, such as conflicting or insufficient evidence. Key findings include the identification of diverse uncertainty sources and the necessity for calibrated uncertainty to guide system actions effectively, ultimately proposing a roadmap for future research in this area.
Uncertainty in multimodal models isn't just a number; it can fundamentally alter decision-making quality under complex evidence conditions.
Multimodal large language models (MLLMs) increasingly answer questions whose correctness depends on visual, textual, temporal, acoustic, document, chart, or embodied evidence. Their failures are therefore not only linguistic. A fluent answer may conceal poor input quality, a perceptual error, weak grounding, conflict between modalities, unstable reasoning, distribution shift, or a question that is not answerable from the supplied evidence. This survey organizes the literature on uncertainty-aware MLLMs around a decision-centered framework: uncertainty sources give rise to observable signals, signals must be calibrated or controlled for risk, and calibrated uncertainty should determine the system action. We review work on token and logit uncertainty, semantic disagreement, perturbation instability, grounding and attribution scores, verbalized confidence, verifier and judge scores, conformal prediction, selective answering, abstention, clarification, retrieval, self-checking, and escalation. The central argument is that uncertainty should not be evaluated only as a confidence number; it should be evaluated by whether it improves behavior under insufficient, conflicting, shifted, or high-risk multimodal evidence. We position this survey against text-only uncertainty and abstention surveys, broad MLLM surveys, MLLM hallucination surveys, and safety-oriented reviews. We conclude with open problems in source-aware decomposition, action-aware benchmarks, calibration under shift, black-box uncertainty estimation, broader modality coverage, reproducible reporting, and human-centered uncertainty communication.
