Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study critically examines the effectiveness of automatic evaluation methods in assessing creativity in text generated by Large Language Models (LLMs). By collecting human evaluations of short stories from both human and AI authors across 11 creativity dimensions, the authors reveal a significant misalignment between human judgments and automated metrics, particularly noting that LLM judges favor AI-generated content. The findings underscore the inadequacy of current evaluation methods, which fail to capture the nuanced and subjective nature of creativity, demonstrating a near-zero correlation with human assessments.
LLMs show a surprising bias towards AI-generated stories, revealing a fundamental flaw in how we evaluate creativity in machine-generated text.
Large Language Models (LLMs) are increasingly capable of generating text that challenges human performance in domains requiring creativity, yet evaluating creativity in LLM-generated content remains a significant challenge. Here, we investigate whether current automatic evaluation methods can reliably capture human judgments of creativity. We collect human evaluations of human- and AI-generated short stories from the WritingPrompts dataset across 11 dimensions of creativity, and compare these judgments with automated objective metrics and LLM-as-a-Judge evaluations. Our experiments reveal substantial misalignment between automatic evaluations and human assessments. In particular, LLM-based judges exhibit a systematic preference for AI-generated stories, consistently favoring their stylistic characteristics over the unpredictability and other qualities of human-authored texts. Furthermore, correlation analyses show that widely used automatic metrics exhibit near-zero alignment with human judgments across both human- and AI-generated stories, suggesting that they fail to capture important dimensions of creativity. These findings highlight fundamental limitations in current approaches to the automatic evaluation of creative text and underscore the difficulty of reducing the multidimensional and subjective nature of creativity to computational metrics.