Search papers, labs, and topics across Lattice.
This study investigates the internal representations of essay quality in eight large language models (LLMs) using two English datasets and one Portuguese dataset. Through methods such as linear probing and neuron-level analyses, the authors reveal that essay quality information is linearly accessible within LLM representations, emerging progressively across layers and demonstrating robustness across different prompts. Notably, the identification of "essay scoring neurons" highlights a correlation with essay scores and sensitivity to interventions, suggesting a structured encoding of quality that enhances the interpretability of LLM-based automated essay scoring systems.
Essay quality is encoded in LLMs as linearly accessible representations, revealing a surprising robustness across prompts and a systematic reliance on deeper layers for longer essays.
Recent advances in Large Language Models (LLMs) have substantially transformed Automated Essay Scoring (AES), yet the internal mechanisms underlying LLM-based scoring remain poorly understood. In this work, we systematically analyze the hidden representations of eight LLMs across two English essay datasets (ASAP++, CSEE) and one Portuguese dataset (ENEM). Using linear probing, cross-prompt generalization, dimensionality reduction, and neuron-level analyses, we find consistent evidence that essay quality information is encoded in a linearly accessible form within LLM representations. These representations emerge progressively across layers, remain robust across prompting strategies, and partially transfer across essay prompts despite differences in scoring rubrics. In addition, nonlinear probes provide only marginal and inconsistent improvements over linear probes, suggesting that most essay quality information is already linearly decodable. We further identify individual ``essay scoring neurons'' whose activations strongly correlate with essay scores and whose behavior is sensitive to targeted intervention. Moreover, the layer-wise distribution of these neurons systematically shifts with essay length, with longer essays relying more heavily on deeper layers. Overall, our findings provide evidence that LLMs encode structured representations related to essay quality and offer new insights into the interpretability of LLM-based AES systems.
