Search papers, labs, and topics across Lattice.
This paper investigates the performance gap between humans and AI models in egocentric action recognition under spatial and spatiotemporal manipulations using the Epic ReduAct dataset. The study compares human and Side4Video model performance on Minimal Identifiable Recognition Crops (MIRCs) and subMIRCs, analyzing the impact of spatial reduction and temporal scrambling. Results indicate humans rely on sparse, semantically critical cues like hand-object interactions and are robust to temporal scrambling when spatial cues are present, while the model depends more on contextual and mid- to low-level features and exhibits class-dependent temporal sensitivities.
Humans nail egocentric action recognition with minimal cues, while AI models often over-rely on context and surprisingly ignore temporal disruptions.
Humans consistently outperform state-of-the-art AI models in action recognition, particularly in challenging real-world conditions involving low resolution, occlusion, and visual clutter. Understanding the sources of this performance gap is essential for developing more robust and human-aligned models. In this paper, we present a large-scale human-AI comparative study of egocentric action recognition using Minimal Identifiable Recognition Crops (MIRCs), defined as the smallest spatial or spatiotemporal regions sufficient for reliable human recognition. We used our previously introduced, Epic ReduAct, a systematically spatially reduced and temporally scrambled dataset derived from 36 EPIC KITCHENS videos, spanning multiple spatial reduction levels and temporal conditions. Recognition performance is evaluated using over 3,000 human participants and the Side4Video model. Our analysis combines quantitative metrics, Average Reduction Rate and Recognition Gap, with qualitative analyses of spatial (high-, mid-, and low-level visual features) and spatiotemporal factors, including a categorisation of actions into Low Temporal Actions (LTA) and High Temporal Actions (HTA). Results show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs, reflecting a strong reliance on sparse, semantically critical cues such as hand-object interactions. In contrast, the model degrades more gradually and often relies on contextual and mid- to low-level features, sometimes even exhibiting increased confidence under spatial reduction. Temporally, humans remain robust to scrambling when key spatial cues are preserved, whereas the model often shows insensitivity to temporal disruption, revealing class-dependent temporal sensitivities.
