Search papers, labs, and topics across Lattice.
This study evaluates the performance of Portugal's AMALIA, a 9B-parameter language model, as a data annotator for coding the moral foundation of authority. While AMALIA shows competitive agreement with human coders, the research reveals significant limitations in its validity, particularly when decomposing prompts into atomic clauses, resulting in only half of its holistic performance being recoverable. The findings suggest that AMALIA relies on surface-level correlates rather than a deep understanding of the theoretical constructs, indicating that national language models may require more rigorous benchmarking beyond mere agreement metrics.
AMALIA's performance reveals that high agreement with human coders does not guarantee valid measurement of complex constructs, highlighting critical flaws in national language model evaluations.
A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration closes that gap, some portability should survive across models and languages; where it does not, the construct-model instrument is the likely locus of failure. We ask whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, especially moral outrage near authority figures. An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, pointing away from the corpus as the main explanation. AMALIA can still screen and pre-code at scale, but it cannot yet measure this construct well enough to stand alone. The study is a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark batteries should test not only agreement with human coders, but the evidential route by which that agreement is warranted.
