Search papers, labs, and topics across Lattice.
This paper introduces DiCoBench, a novel benchmark specifically designed to assess fine-grained perception in Multimodal Large Language Models (MLLMs) using high-resolution images and implicit visual cues. By categorizing tasks into Differential and Commonality Visual Cues and employing a multiple-choice format, the benchmark effectively addresses the limitations of existing evaluation methods that rely on low-resolution inputs and explicit textual cues. The evaluation of 18 MLLMs reveals a significant performance gap compared to human accuracy, highlighting the challenges models face in capturing micro-scale details, thus underscoring the need for improved perception capabilities in AI systems.
MLLMs struggle to match human accuracy in fine-grained perception, with a striking performance gap revealed by the new DiCoBench benchmark.
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive fine-grained perception capabilities. However, existing benchmarks predominantly rely on explicit textual cues or low-resolution inputs, failing to evaluate a model's ability to autonomously perceive implicit visual cues in high-resolution. To bridge this gap, we introduce DiCoBench, a comprehensive, multi-image high-resolution benchmark designed for cross-image fine-grained perception. DiCoBench consists of 765 meticulously curated samples categorized into two progressive tracks: Differential Visual Cues and Commonality Visual Cues, covering 8 distinct perception tasks. By formulating the benchmark as a multiple-choice question task and utilizing high-resolution imagery (approaching 2K), we eliminate evaluation metric bias and pose a substantial challenge to current state-of-the-art MLLMs. Our extensive evaluation of 18 diverse MLLMs reveals a striking performance gap compared to human accuracy (98.3\%), with top-performing models struggling significantly with micro-scale detail capture. We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.
