Search papers, labs, and topics across Lattice.
This study evaluates the integrity of language-model honesty assessments by shifting the focus from model outputs to the evaluation instrument itself, utilizing a text-adventure game where the game engine determines quest outcomes. By manipulating decision rules and recording them prior to evaluation, the authors reveal that instrument choices significantly influence measured honesty, with a notable drop in strong claims when expanding verdict options. The findings highlight the instability of verdict distributions across repeated runs and suggest a need for a four-check integrity protocol for evaluation instruments to ensure reliable assessments.
Expanding the verdict grammar in honesty evaluations can drastically alter the perceived reliability of language models, with strong claims plummeting from 38 to just 7 out of 40.
Evaluations of language-model honesty read the model's verdicts as evidence about the model. We test the instrument instead. We built a text-adventure world where the game engine, not any model, knows whether the quest can be completed. A language model plays under a budget and must eventually declare its quest complete, unreachable, or not yet decidable; the engine scores every verdict. Decision rules were recorded before results were read, and run artifacts bind the revisions they executed; the strength of preregistration varies by series and is disclosed. With the player held fixed, instrument choices substantially changed measured behavior. On four byte-identical anchors, expanding a two-verdict grammar to three verdicts moved strong claims from 38/40 to 7/40, while the new incomplete verdict took 28/40 outcomes; across series 2, 93/158 valid games ended incomplete. One sentence disclosing the success criterion took matched-instance false verdicts from 18/59 to 0/58, through fewer decision points and cleaner decisions. Repeated runs of one fixed configuration produced non-stable verdict distributions on 3 of 4 instances: single runs report samples as dispositions. A formally preregistered narrative-register gradient was falsified; two post-hoc, hypothesis-generating patterns remain: register presence roughly doubled strong claims, and budget rendering moved verdicts more than register content (.383 meter vs .150 lantern). The narrator compressed abundant budgets toward scarcity landmarks, yet the registered mediation test returned a null. We propose a four-check integrity protocol for eval instruments.
