Search papers, labs, and topics across Lattice.
This study assesses the effectiveness of GPT-4o-mini in scoring open-ended music analysis essays by comparing its outputs to teacher mean scores across four dimensions: Harmony, Form, Reasoning, and Terminology. Using three distinct prompting strategies鈥攆ew-shot prompting with chain-of-thought reasoning, retrieval-augmented generation, and self-consistency鈥攖he authors found that the few-shot prompting method yielded the highest agreement with teacher scores. The results highlight the importance of strategy-specific calibration and dimension-level validation for reliable automated scoring in music analysis contexts.
Few-shot prompting with chain-of-thought reasoning outperforms other strategies, achieving the best alignment with teacher assessments in music analysis scoring.
Scoring open-ended music analysis responses is time-consuming and requires nuanced judgments of harmonic knowledge and formal understanding. This study evaluates the validity and repeatability of GPT-4o-mini for rubric-based scoring of music analysis essays, using teacher mean scores as the benchmark. A dataset of 300 university-level student responses was scored by teachers on four dimensions: Harmony, Form, Reasoning, and Terminology. GPT-4o-mini scored the same responses using three prompting strategies: few-shot prompting with chain-of-thought reasoning (Fs+CoT), retrieval-augmented generation (RAG), and self-consistency based on five internal generations per administration (SC). Each strategy was administered three times with the model, prompt, rubric, and response held constant. Single-pass scores represented an operational scoring condition, whereas median aggregation across three runs was used to examine robustness. Agreement with teacher mean scores was evaluated using correlation, intraclass correlation, Krippendorff's alpha, quadratic weighted kappa, and scoring error indices. Fs+CoT showed the strongest agreement with teacher mean scores in both single-pass scoring and median aggregation. RAG showed systematic over-scoring, whereas SC produced highly repeatable scores but weaker individual-level agreement. Dimension-level analyses showed that scoring performance varied across rubric components, with Terminology generally showing weaker agreement than Reasoning. These findings indicate that GPT-4o-mini can generate stable scores for complex music analysis responses, but prompting strategies produce distinct scoring profiles. Operational use therefore requires strategy-specific calibration, dimension-level validation, and continued human oversight.
