Search papers, labs, and topics across Lattice.
This paper introduces Poller, a novel method that utilizes large language models (LLMs) to evaluate poetry understanding by simulating the perspective of the poem's author. By conducting extensive experiments across eight specialized dimensions, the authors demonstrate that Poller significantly reduces evaluation error compared to traditional methods, achieving up to 94.55% error reduction in assessing rhetorical techniques. This approach not only enhances the efficiency of poetry evaluation but also maintains a high level of accuracy, bridging the gap between automated processes and human judgment.
Poller achieves unprecedented accuracy in poetry evaluation, reducing error rates by up to 94.55% compared to traditional methods.
Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging large language models (LLMs) to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.
