Search papers, labs, and topics across Lattice.
The paper introduces SIEVES, a selective prediction method for multimodal LLMs that improves out-of-distribution (OOD) generalization by explicitly scoring the quality of localized visual evidence provided by the reasoner. SIEVES learns to estimate the quality of visual localization, enabling the model to abstain from answering when the evidence is deemed insufficient, thereby improving coverage while maintaining a user-defined risk level. Experiments on five challenging OOD benchmarks demonstrate that SIEVES achieves up to a three-fold increase in coverage compared to baselines, and can transfer to proprietary reasoners without access to their weights or logits.
Achieve 3x better coverage on out-of-distribution visual question answering by explicitly scoring the quality of visual evidence, even when using black-box models like Gemini-3-Pro.
Multimodal large language models (MLLMs) achieve ever-stronger performance on visual-language tasks. Even as traditional visual question answering benchmarks approach saturation, reliable deployment requires satisfying low error tolerances in real-world out-of-distribution (OOD) scenarios. Precisely, selective prediction aims to improve coverage, i.e. the share of inputs the system answers, while adhering to a user-defined risk level. This is typically achieved by assigning a confidence score to each answer and abstaining on those that fall below a certain threshold. To enable reliable generalization, we require reasoner models to produce localized visual evidence while answering, and design a selector that explicitly learns to estimate the quality of the localization provided by the reasoner. We show that SIEVES (Selective Prediction through Visual Evidence Scoring) improves coverage by up to three times on challenging OOD benchmarks (V* Bench, HR-Bench-8k, MME-RealWorld-Lite, VizWiz, and AdVQA), compared to non-grounding baselines. Beyond better generalization to OOD tasks, the design of the SIEVES selector enables transfer to proprietary reasoners without access to their weights or logits, such as o3 and Gemini-3-Pro, providing coverage boosts beyond those attributable to accuracy alone. We highlight that SIEVES generalizes across all five tested OOD datasets and reasoner models (Pixel-Reasoner, o3, and Gemini-3-Pro), without benchmark- or reasoner-specific training or adaptation.
