Search papers, labs, and topics across Lattice.
This study investigates the impact of probability calibration on evaluator feedback loops in large language model (LLM) agents, addressing the issue of evaluator preference coupling that leads to biased behavior adaptation. By conducting a within-subjects experiment with DeepSeek-V4-Pro and GLM5.2, the authors demonstrate that applying confidence calibration to pairwise judgments significantly reduces the coupling coefficient by 20-49% and the Jensen-Shannon divergence by 45-67%. The findings suggest that calibrated TTRL can serve as an effective and lightweight mitigation strategy for improving LLM performance in evaluative roles.
Calibrating evaluator feedback can slash preference coupling in LLMs by nearly half, enhancing their decision-making integrity.
When large language model (LLM) agents adapt their behavior through evaluator feedback, systematic evaluator biases propagate into the agent's learned strategy distribution - a phenomenon termed evaluator preference coupling. Prior work has documented this coupling and established a diagnostic framework (EPC) to measure it, but has not investigated whether calibration techniques can mitigate the effect. We present the first study of evaluator calibration as mitigation: applying probability calibration to the evaluator's pairwise judgments to reduce spurious preference propagation. In a controlled within-subjects experiment (N=5) comparing standard binary TTRL (win/loss) with confidence-calibrated TTRL (probability-weighted updates) using DeepSeek-V4-Pro as executor and GLM5.2 as evaluator, we find that calibration reduces the coupling coefficient gamma by 20-49% and Jensen-Shannon divergence by 45-67%. A symmetric-LR control confirms the effect is not due to reduced update asymmetry. We release the calibrated TTRL protocol and recommend it as a lightweight mitigation for LLM-as-judge deployment pipelines.
