Search papers, labs, and topics across Lattice.
This study investigates the alignment between human and large language model (LLM) evaluations of non-native Japanese writing, focusing on fluency, status, and solidarity. It reveals that while LLMs replicate the bias of human raters鈥攚ho rated L2 texts significantly lower across all dimensions鈥攖hey also exhibit notable divergences, particularly in underestimating the solidarity gap and differentiating among learner backgrounds. These findings highlight the potential risks non-native speakers face in high-stakes evaluations and suggest that the language attitudes framework can serve as an effective tool for auditing LLM biases beyond English.
LLMs mirror human biases against non-native Japanese speakers, but with critical underestimations that could exacerbate inequities in high-stakes evaluations.
Large language models (LLMs) increasingly evaluate human writing in high-stakes domains such as hiring and academic assessment, putting non-native speakers at particular risk. Drawing on the language attitudes framework, we compared human and LLM evaluations of parallel L1- and L2-written Japanese emails on three dimensions: fluency, status, and solidarity. Japanese raters rated L2 texts significantly lower on all three dimensions, with a fluency gap roughly twice the size of the status and solidarity gaps. Six LLM judges reproduced the direction of this bias, and five reproduced its ordering across dimensions. The models diverged from humans in two ways: all understated the solidarity gap, the most socially grounded dimension, and all differentiated among learner L1 backgrounds where humans did not. LLM judges thus reproduce native speakers' language attitudes in a structured yet attenuated form, and the language attitudes framework offers a ready-made yardstick for auditing them beyond English.
