Search papers, labs, and topics across Lattice.
This study investigates the impact of evidence locking on the evaluation process of large language models (LLMs) acting as judges, comparing it to standard pairwise and structured one-call judging methods. The findings reveal that evidence locking, which involves persisting evidence as the exclusive input for subsequent judgments, leads to a significant decrease in agreement with human preferences and increased inconsistency in answer ordering. Ultimately, while evidence locking may enhance auditability, it detracts from the accuracy of judgments, suggesting that it should not substitute the original answers during decision-making.
Evidence locking in LLM evaluations can degrade judgment accuracy by up to 6 percentage points, challenging the assumption that preserving evidence enhances decision-making.
LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.
