Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a cognitively motivated framework that dissects metaphor explanation quality into six dimensions, addressing the limitations of traditional holistic evaluations. Through an extensive annotation study involving 11,200 ratings, the authors reveal that explanation quality is multidimensional, with systematic annotator disagreement and a shared cluster of dimensions. Additionally, they demonstrate that an automatic evaluation pipeline can effectively capture aspects of this multidimensional structure, highlighting the need for richer diagnostic insights in evaluating metaphor explanations.
Metaphor explanations are not just about quality; they reveal a complex, multidimensional landscape that challenges traditional evaluation methods.
Current evaluation of metaphor explanations relies mainly on holistic quality ratings, revealing little about how explanation quality is structured or where human judgments agree and diverge. We introduce a cognitively motivated framework that decomposes metaphor explanation quality into six theoretically grounded dimensions. In a dense annotation study (11,200 ratings), we find that: {\bfseries(i)} explanation quality is genuinely multidimensional; {\bfseries(ii)} annotator disagreement is systematic rather than random; and {\bfseries(iii)} the six dimensions collapse into a shared cluster and two independent axes of judgment. An exploratory feasibility study further shows that a standard automatic evaluation pipeline can recover parts of this structure, predicting the most discriminative dimensions well while its errors correlate human (dis)agreement. Together, these results suggest that multidimensional evaluation offers richer diagnostic insight than holistic ratings, and that automatic evaluators for open-ended generation tasks should be judged on how well they preserve the structure of human judgment.