Search papers, labs, and topics across Lattice.
This paper introduces LL-Bench, a novel benchmark designed to evaluate large-scale generative models on low-level vision tasks, addressing a critical gap in understanding their pixel-wise control capabilities. The benchmark includes a substantial dataset of 2,469 real-world degraded images across 16 degradation tasks, alongside 28,919 restored images from various models, annotated with expert-level human preferences. Key findings reveal significant performance boundaries and failure modes of generative models, and the authors propose LL-Score, an MLLM-based evaluator that aligns restoration quality assessment with human preferences, outperforming existing metrics.
Large-scale generative models struggle with low-level vision tasks, revealing critical performance gaps that conventional metrics fail to capture.
Large-scale generative models have demonstrated remarkable capabilities across image generation and editing tasks. However, their performance in low-level vision tasks, which require pixel-wise control, remains insufficiently studied. To address this gap, we introduce \textbf{LL-Bench}, a comprehensive \textbf{Benchmark} for evaluating the capabilities of large-scale generative models on \textbf{L}ow-\textbf{L}evel vision tasks. The benchmark comprises 2,469 real-world degraded images covering 16 low-level degradation tasks, and 28,919 restored images produced by 10 state-of-the-art large-scale generative models and 21 conventional restoration models, which are annotated with 152,020 expert-level pairwise human preferences and 28,334 quality scores. Built upon LL-Bench, we present a systematic diagnosis that reveals the performance boundaries and unique failure modes of large-scale generative models across diverse low-level vision tasks, compared with conventional representative restoration approaches. Moreover, we investigate the effectiveness of current quality evaluation metrics on LL-Bench, which exhibit significant discrepancy with human ratings. To better align restored-image quality assessment with human preferences, we further propose \textbf{LL-Score}, an MLLM-based evaluator that captures both restoration quality and hallucination existence. Extensive experiments demonstrate that LL-score not only outperforms existing image quality assessment metrics, but also serves as a promising reward model for training generative models on low-level vision tasks.
