Search papers, labs, and topics across Lattice.
This paper introduces the C4 evaluation framework designed to assess the cross-concept understanding capabilities of multi-layered language models (MLLMs) through a structured approach to encoding and decoding creative concepts, specifically using Chengyu (Chinese idioms). By constructing a comprehensive evaluation set (C4-Eval) with 184 synthetic items and 37 human-created figures, the authors reveal that the best-performing closed models achieve only 50.7% accuracy in recovering intended meanings, highlighting significant limitations in current MLLM creative decoding abilities. The findings underscore the necessity for improved methodologies in evaluating and enhancing the creative capacities of AI systems, particularly in complex conceptual domains.
Current MLLMs struggle with creative decoding, achieving only 50.7% accuracy in understanding cross-concept relations, revealing a critical gap in their cognitive capabilities.
Creative capabilities of MLLMs matter in design, communication, education, and human--AI collaboration, yet remain difficult to evaluate because explicit targets and reward signals are scarce compared with accuracy-oriented tasks. Cross-concept understanding is a core cognitive capacity underlying receptive creativity. It enables a perceiver to recover intended meaning from non-obvious but meaningful conceptual relations. We operationalize item construction as cross-concept encoding and model inference as cross-concept decoding. We introduce C4, a cognition-inspired evaluation framework for Chengyu (Chinese idiom)-based Cross-Concept Creativity. Its encoding component maps target slots to imageable substitute concepts along bridge paths in a manually annotated and third-party-reviewed cross-concept network, enabling batch generation with explicit structure, difficulty indexed by bridge count and depth, and exact answers. Using this framework, we instantiate the C4 Evaluation Set (C4-Eval), comprising 184 synthetic items and 37 human-created cross-concept chengyu figures collected from online sources. We manually construct and review cross-concept relations, bridge paths, and reasoning processes for the collected figures. Each C4-Eval item is instantiated in five task settings, yielding 884 primary answer-recovery cases. Across ten evaluated MLLMs, the strongest closed models reach 50.7% and 48.0% primary accuracy, while open-source models remain substantially lower. Candidate constraints improve accuracy sharply, but bridge hints and explanation requests provide only modest gains. These results expose a substantial gap in how current MLLMs decode creatively encoded meaning through cross-concept relations. The code is in the supplementary material.
