Search papers, labs, and topics across Lattice.
This paper investigates the feasibility of LLM-assisted Behavioral Profile (BP) annotation by decomposing the task into individual annotation skills defined by schema files, decision rules, and examples. Using a dataset of 3,134 Chinese metaphorical color-term derivatives and a 14-feature BP schema, they evaluated GPT-5.4 and open-source models on a subset of operable skills identified through human annotation. The results demonstrate that while GPT-5.4 can reliably execute a subset of these skills, its performance is selective and better understood as an independent annotator rather than a direct substitute for human annotators.
LLMs can handle some behavioral annotation tasks with surprising reliability, but only if you break the problem down into clearly defined, independently operable skills.
Behavioral Profile (BP) annotation is difficult to automate because it requires simultaneous coding across multiple linguistic dimensions. We treat BP annotation as a bundle of annotation skills rather than a single task and evaluate LLM-assisted BP annotation from this perspective. Using 3,134 concordance lines of 30 Chinese metaphorical color-term derivatives and a 14-feature BP schema, we implement a skill-file-driven pipeline in which each feature is externally defined through schema files, decision rules, and examples. Two human annotators completed a two-round schema-only protocol on a 300-instance validation subset, enabling BP skills to be classified as directly operable, recoverable under focused re-annotation, or structurally underspecified. GPT-5.4 and three locally deployable open-source models were then evaluated under the same setup. Results show that BP annotation is highly heterogeneous at the skill level: 5 skills are directly operable, 4 are recoverable after focused re-annotation, and 5 remain structurally underspecified. GPT-5.4 executes the retained skills with substantial reliability (accuracy = 0.678, \k{appa} = 0.665, weighted F1 = 0.695), but this feasibility is selective rather than global. Human and GPT difficulty profiles are strongly aligned at the skill level (r = 0.881), but not at the instance level (r = 0.016) or lexical-item level (r = -0.142), a pattern we describe as shared taxonomy, independent execution. Pairwise agreement further suggests that GPT is better understood as an independent third skill voice than as a direct human substitute. Open-source failures are concentrated in schema-to-skill execution problems. These findings suggest that automatic annotation should be evaluated in terms of skill feasibility rather than task-level automation.
