Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This paper introduces PRISM, a method for calibrating black-box LLM judges by measuring score changes in response to counterfactual probes that emphasize or de-emphasize specific attributes, treating these changes as a surrogate Jacobian. By learning a correction from human labels using ridge regression on this surrogate Jacobian, PRISM significantly improves agreement with human rankings on summarization and story evaluation datasets without requiring rubric-level ratings. The learned coefficients offer insights into judge-specific biases and correction strategies.
LLM judges can be recalibrated to align with human preferences by probing their sensitivity to evaluative biases, even without explicit rubric criteria.
Large language models are increasingly used as judges to reduce reliance on costly human evaluation, yet their scores can systematically disagree with human judgments. We ask whether responses to an explicitly imposed evaluative bias can help correct a judge's implicit scoring biases. We introduce PRISM (Probe-based Residual Inference of Scoring Misalignment), which holds the evaluated content fixed and measures score changes when attributes are emphasized or de-emphasized. A local sensitivity model motivates representing these responses as a surrogate Jacobian. Ridge regression on human labels learns a correction to the baseline score. The resulting scoring protocol is rubric-free: it uses overall scores without requesting criterion-level ratings. In the main Jev and Sol comparisons on SummEval summarization and HANNA story evaluation, PRISM and its learned-baseline variant significantly improve agreement with human rankings over uncalibrated scores while remaining competitive with rubric-based methods. A controlled SummEval comparison yields similar agreement for rubric-derived quality probes and style probes, supporting calibration without reproducing rubric criteria. The learned coefficients reveal shared and opposing correction directions across judges.
MIT CSAIL