Search papers, labs, and topics across Lattice.
This paper introduces HoloCount, a comprehensive benchmark designed to evaluate the counting capabilities of Multimodal Large Language Models (MLLMs) across three hierarchical levels: Semantic Counting, Analytical Counting, and Robustness Testing. The authors highlight a significant performance gap where even leading MLLMs struggle with complex analytical reasoning and adverse scenarios, revealing persistent numerical hallucinations in their outputs. By systematically assessing over 20 state-of-the-art models, the study provides critical insights into the limitations of current MLLMs and lays the groundwork for future improvements in multimodal intelligence.
Even top-tier MLLMs can falter dramatically in counting tasks that require complex reasoning, revealing a critical gap in multimodal intelligence.
Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.
