Search papers, labs, and topics across Lattice.
This paper introduces an automated, domain-agnostic framework for evaluating the creativity of large language models (LLMs) across open-ended tasks, addressing the limitations of existing metrics that are often tied to specific domains. By employing semantic entropy to measure divergent creativity and a novel retrieval-based multi-agent judge framework for convergent creativity, the authors achieve over 60% efficiency improvement in task fulfillment assessments. The framework is validated across diverse domains, revealing significant insights into how various model properties influence creative performance, thereby establishing a reproducible standard for automated creativity evaluation in AI.
Automated creativity evaluation reveals that LLMs can achieve significant improvements in task fulfillment efficiency, transforming how we assess AI creativity across diverse applications.
Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.
