Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the inconsistencies in evaluating Knowledge Graph Completion (KGC) models caused by reliance on isolated rank-based metrics, which can yield conflicting results across datasets. By reframing KGC evaluation as a Multi-Criteria Decision-Making (MCDM) problem, the authors conduct a meta-analysis of seven aggregators using five rigorous tests to assess their performance. The findings reveal that the Z-score aggregator provides the most balanced evaluation, ranking DualE highest for tail prediction and FMS highest for relation prediction, thereby enhancing the reliability of KGC model comparisons.
Z-score emerges as the most reliable aggregator for KGC model evaluation, challenging the status quo of metric-based assessments.
Evaluating Knowledge Graph Completion (KGC) models remains challenging because standard assessment relies on isolated rank-based metrics such as MRR, Hits$@$k, and Mean Rank, which often produce conflicting model orderings across datasets. A model that leads on MRR may trail on Hits@1, and strong performance on one dataset may not generalize to another. This fragmentation hinders comparison, enables selective reporting, and obscures real progress. We reframe KGC evaluation as a Multi-Criteria Decision-Making (MCDM) problem and present a meta-analysis of seven aggregators across five tests: consistency, cross-dataset stability, metric independence, robustness under noise, and generalizability. Each test is averaged over leave-one-model-out (LOMO) and leave-one-group-out (LOGO) removals so that reliability reflects aggregator behavior across diverse model subsets. Across tail $(h,r,?)$ and relation $(h,?,t)$ prediction, Pareto-optimal analysis identifies Z-score as the most balanced aggregator, which ranks DualE highest for tail prediction and FMS (Flow-Modulated Scoring) highest for relation prediction. A test-sensitivity analysis using the same removals shows that consistency and stability are largely removal-invariant, while generalizability and independence are the most sensitive. The framework resolves evaluation inconsistencies and offers evidence-based guidance for aggregator selection and model benchmarking in KGC.