Search papers, labs, and topics across Lattice.
This paper introduces FG-BMK, a novel benchmark designed to evaluate Large Vision-Language Models (LVLMs) on fine-grained image tasks, comprising over 1 million questions and 280,000 images across various domains. The study reveals that current LVLMs struggle with fine-grained recognition due to intertwined issues in visual representation, semantic grounding, and modality alignment. By diagnosing these failures, the authors provide actionable insights for improving model training and data construction, ultimately enhancing the reliability of LVLMs in fine-grained visual tasks.
Current LVLMs are inadequate at fine-grained image recognition, revealing critical bottlenecks in visual and semantic processing that need urgent attention.
Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal perception and reasoning capabilities. While numerous benchmarks have evaluated LVLMs from holistic or task-specific perspectives, their capabilities on fine-grained image tasks-fundamental to computer vision-remain insufficiently understood. To address this gap, we introduce FG-BMK, a comprehensive fine-grained evaluation benchmark containing 1.01 million questions and 0.28 million images, covering diverse scenarios from common object-centric domains to specialized domains. FG-BMK jointly evaluates dialogue-level fine-grained semantic recognition and feature-level visual discriminability through human-oriented and machine-oriented paradigms, enabling diagnostic analysis of whether LVLM failures arise from insufficient visual representations, weak visual-to-semantic grounding, or limited fine-grained knowledge. Through extensive experiments on a diverse set of representative LVLMs/VLMs, we find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge. We further analyze training design factors for improving fine-grained capabilities and examine how visual and linguistic perturbations affect LVLM predictions. These findings provide diagnostic insights into the limitations of current LVLMs and offer guidance for future data construction and model design in developing more reliable LVLMs for fine-grained visual tasks. Our code is open-source and available at https://fg-bmk.github.io/.
