Search papers, labs, and topics across Lattice.
This paper introduces EgoGapBench, a benchmark designed to evaluate Egocentric Action Selection (EAS) in multi-agent scenarios, addressing the limitations of existing egocentric benchmarks that rely solely on first-person-view data. The findings reveal that while humans perform reliably on this benchmark, both open-source and proprietary MLLMs struggle significantly, often selecting actions of other agents instead of appropriate actions from their own perspective. Fine-tuning on EgoGapBench data improves model performance but does not achieve human-level accuracy, highlighting the complexity of EAS beyond mere scene understanding.
MLLMs falter in egocentric action selection, consistently opting for actions of visible agents instead of their own, revealing a critical gap in current training paradigms.
Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.
