Search papers, labs, and topics across Lattice.
This paper conducts a systematic analysis of the evaluation pitfalls in multimedia event extraction, identifying three critical sources of inconsistency: data processing, task assumptions, and evaluation settings. Through controlled experiments, the authors reveal that even minor changes in evaluation criteria can significantly skew performance metrics, often leading to an overestimation of models' capabilities in grounding real-world events across different modalities. The findings underscore the urgent need for standardized evaluation practices to ensure reliable comparisons and assessments in this evolving field.
Minor tweaks in evaluation can drastically inflate performance metrics, revealing the fragility of current multimedia event extraction assessments.
Multimedia event extraction aims to jointly identify events and their arguments across multiple modalities, such as text and images, to support more comprehensive event understanding. While recent work reports steady and substantial progress, the reliability and comparability of these results critically depend on consistent and rigorous evaluation. In this work, we present the first systematic analysis of evaluation pitfalls in multimedia event extraction and identify three major sources of issues: inconsistent data processing, inconsistent task assumptions, and overly relaxed evaluation settings. We demonstrate, through a series of controlled experiments under a strict evaluation framework, that minor evaluation choices can cause large performance variations and lead to overestimation of a model's ability to ground real-world events across modalities. Our findings highlight the need for comparable evaluation standards and encourage a shift toward more rigorous evaluation in multimedia event extraction.
