Search papers, labs, and topics across Lattice.
This paper introduces CreaEval, an innovative automated creativity evaluator designed to address biases in LLM evaluations of complex multi-step creativity tasks (CGPST). By decoupling the evaluation process into two distinct phases鈥擬emory-augmented Analysis and Evidence-based Judging鈥擟reaEval enhances the reliability of creativity assessments by utilizing structured evidence while mitigating issues like verbosity and leniency bias. Experimental results indicate that CreaEval significantly outperforms existing methods, achieving an average performance improvement of 22.74% across various tasks, underscoring its effectiveness and generalizability.
CreaEval's dual-phase approach not only mitigates biases in LLM evaluations but also boosts performance by over 22% in complex creativity tasks.
Automated evaluation of creativity tasks remains challenging for LLM-as-a-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias. Such limitations are particularly evident in Contextually-Grounded and Procedurally-Structured Tasks (CGPST), a complex multi-step creativity task where inter-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments. Existing approaches either rely on task-specific training or directly apply LLM-as-a-Judge, both of which struggle to ensure reliable evaluation under such complexity. To bridge these gaps, we propose CreaEval, an automated creativity evaluator for CGPST that decouples typical LLM-as-a-Judge into analysis and judging. Correspondingly, CreaEval involves two critical phases: Memory-augmented Analysis, a SoT-LLM converts multi-step responses into structured evaluation evidence, incorporating cross-step memory; and Evidence-based Judging, a Judge-LLM uses the extracted evidence for judging without accessing raw responses. Comprehensive experiments show that CreaEval achieves an average performance improvement of 22.74% over the second-best baselines across CGPST and two classic simple creativity tasks, demonstrating its generalizability. The code is available at https://github.com/Jaong/CreaEval.
