Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
DistortBench, a new diagnostic benchmark, assesses VLMs' ability to identify image distortion types and severity across 27 distortions, 6 perceptual categories, and 5 severity levels using a four-choice question format. Evaluating 18 VLMs reveals that even the best models achieve only 61.9% accuracy, falling short of the human majority-vote baseline of 65.7%, highlighting a significant gap in low-level perceptual understanding. Further analysis exposes weak scaling with model size and performance variations across model families, suggesting areas for targeted improvement.
VLMs, despite excelling at high-level tasks, struggle to identify even basic image distortions, performing worse than humans on a new benchmark designed to test this fundamental visual skill.
Vision-language models (VLMs) are increasingly used in settings where sensitivity to low-level image degradations matters, including content moderation, image restoration, and quality monitoring. Yet their ability to recognize distortion type and severity remains poorly understood. We present DistortBench, a diagnostic benchmark for no-reference distortion perception in VLMs. DistortBench contains 13,500 four-choice questions covering 27 distortion types, six perceptual categories, and five severity levels: 25 distortions inherit KADID-10k calibrations, while two added rotation distortions use monotonic angle-based levels. We evaluate 18 VLMs, including 17 open-weight models from five families and one proprietary model. Despite strong performance on high-level vision-language tasks, the best model reaches only 61.9% accuracy, just below the human majority-vote baseline of 65.7% (average individual: 60.2%), indicating that low-level perceptual understanding remains a major weakness of current VLMs. Our analysis further reveals weak and non-monotonic scaling with model size, performance drops in most base--thinking pairs, and distinct severity-response patterns across model families. We hope DistortBench will serve as a useful benchmark for measuring and improving low-level visual perception in VLMs.