Search papers, labs, and topics across Lattice.
This paper investigates the stability of LLM judges compared to human examiners when scoring Thai bar exam essays, using an identical-inputs protocol with 26 LLMs and 3 human experts. The key finding is that while all raters agree on rubric-prescribed axes, LLMs overwhelmingly converge on the majority human interpretation when the rubric is silent, failing to reproduce a minority human reading. This asymmetry in LLM judgment raises concerns about using human agreement as the sole benchmark for LLM judge selection.
LLMs may excel at mimicking majority human opinions, but they struggle to reproduce valid minority interpretations, revealing a critical flaw in using human consensus as the gold standard for LLM evaluation.
Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling as evidence of judge stability. We test both assumptions on the Thai bar examination through an identical-inputs protocol: three Bar Council-trained examiners (A, B, C) and a 26-LLM judge panel score the same 15 cross-graded answers from the same four inputs (question, official Bar Council grading regulation, gold answer, candidate answer). The headline finding is asymmetric. On 10 of 15 cells where the rubric prescribes both axes, all 29 raters converge in a tight band: panel agreement is universal. On the remaining 5 cells where the rubric does not prescribe how to grade a correct final answer that omits a decisive statutory citation, the human panel splits between two coherent readings (B/C majority at the upper rubric band, score $6$--$8$; A minority at the lower band, score $1$--$2$). The LLM judge population does not split symmetrically: 22 of 26 LLMs score in or near B/C's contested band, 3 sit in the regulation-silent middle gap, and only 1 (GPT-5.4 Nano) approaches A's band without consistently scoring within it. \emph{Zero LLMs in our 26-judge panel reproduce the minority human reading on the contested cells.} The B/C-direction cluster spans every model size, vendor, and price tier we tested. An instrumented three-LLM anchor sub-panel (Claude 4.6 Opus, Gemini 3.1 Pro, GPT-5.4 Pro) carries determinism probes, input ablations, and bootstrap CIs, and reaches anchor panel $伪= 0.77$ on the 15 cells against human-panel $伪= 0.36$. The high LLM-panel $伪$ reflects systematic convergence on the majority reading rather than balanced reproduction of both readings; a benchmark that selects its LLM judge by maximising agreement with a human reference panel will inherit this asymmetry by construction.
