Search papers, labs, and topics across Lattice.
This paper identifies and analyzes Perceptual Judgment Bias in multimodal large language models (MLLMs), where these models favor plausible narratives over accurate visual evidence. To mitigate this bias, the authors introduce the Perceptually Perturbed Judgment Dataset, which facilitates the creation of counterfactual responses that highlight perceptual errors, and develop a unified training framework that integrates a structured GRPO-based reward with a batch-ranking objective. Experimental results demonstrate significant improvements in perceptual fidelity and ranking coherence, showcasing a scalable method for enhancing the reliability of MLLMs as evaluators.
MLLMs often prioritize plausible narratives over accurate visual cues, but a new training framework can significantly enhance their perceptual reliability.
Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.
