Search papers, labs, and topics across Lattice.
This paper investigates the hidden reliability issues in multimodal agentic search systems, introducing a six-category taxonomy of silent failures that can compromise the effectiveness of these systems. By developing a trajectory-level diagnostic pipeline, the authors evaluate both answer correctness and evidence-grounding quality, revealing that traditional accuracy metrics often overestimate true performance. Experiments demonstrate that silent failures are not only prevalent but also capability-dependent, shifting rather than vanishing across different models.
Surface accuracy metrics can mislead researchers about the true reliability of multimodal search systems, with silent failures lurking beneath the surface.
Multimodal agentic search systems increasingly rely on external tools to answer knowledge-intensive visual questions. However, existing evaluations mainly focus on final-answer accuracy and may miss failures in the search trajectory. In this work, we study such hidden reliability issues as silent failures. We introduce a six-category taxonomy covering modality shortcuts, phantom grounding, wrong-evidence-right-answer cases, over-retrieval laundering, cross-modal contradiction, and provenance hallucination. Based on this taxonomy, we build a trajectory-level diagnostic pipeline that evaluates both answer correctness and evidence-grounding quality under a unified ReAct-style scaffold. Experiments on MMSearch-Plus trajectories across four frontier multimodal models show that surface accuracy consistently overestimates true trajectory-level correctness. We further use cross-judge validation, blank-image stress tests, and tool ablations to show that silent failures are capability-dependent and often shift rather than disappear. Home-page: https://github.com/DingWu1021/silent-failures-multimodal-agentic-search
