Search papers, labs, and topics across Lattice.
EC-Bench is introduced as a new benchmark for evaluating enumeration, counting, and temporal evidence grounding in videos longer than 30 minutes, addressing the limitations of existing benchmarks focused on short clips. The benchmark comprises 152 videos and 1,699 queries with explicit evidence spans, designed to test long-range temporal reasoning in multimodal large language models (MLLMs). Experiments across 22 MLLMs reveal significant performance gaps compared to human baselines, with the best model achieving only 29.98% accuracy on Enumeration and 23.74% on Counting, highlighting the challenges of long-form quantitative video reasoning.
Current multimodal LLMs struggle to count objects and ground evidence in videos longer than 30 minutes, achieving only ~25% accuracy compared to human performance on a new benchmark.
Counting in long videos remains a fundamental yet underexplored challenge in computer vision. Real-world recordings often span tens of minutes or longer and contain sparse, diverse events, making long-range temporal reasoning particularly difficult. However, most existing video counting benchmarks focus on short clips and evaluate only the final numerical answer, providing little insight into what should be counted or whether models consistently identify relevant instances across time. We introduce EC-Bench, a benchmark that jointly evaluates enumeration, counting, and temporal evidence grounding in long-form videos. EC-Bench contains 152 videos longer than 30 minutes and 1,699 queries paired with explicit evidence spans. Across 22 multimodal large language models (MLLMs), the best model achieves only 29.98% accuracy on Enumeration and 23.74% on Counting, while human performance reaches 78.57% and 82.97%, respectively. Our analysis reveals strong relationships between enumeration accuracy, temporal grounding, and counting performance. These results highlight fundamental limitations of current MLLMs and establish EC-Bench as a challenging benchmark for long-form quantitative video reasoning.
