Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces CorrGap, a novel metric to quantify inter-dimension dependence in LLM evaluations of open-ended text, revealing that LLM judges often rely on non-target dimensions when assessing quality. The authors propose DimCheck, a method that iteratively removes unrelated evidence from chain-of-thought (COT) outputs to mitigate this dependence, demonstrating its effectiveness across three LLMs and four evaluation tasks. Key findings indicate that DimCheck not only reduces inter-dimension dependence but also allows smaller LLMs to achieve performance levels comparable to larger models at a fraction of the inference cost.
Inter-dimension dependence in LLM evaluations is pervasive, but a new method can significantly reduce this bias while enabling smaller models to perform competitively.
LLM-as-a-judge methods are widely used for evaluating the quality of generated open-ended text. Such evaluations are generally multi-dimensional, since the error patterns in texts can be different for different dimensions. Therefore, reliable LLM judges should evaluate each target dimension independently. To quantify the extent to which LLM judges depend on non-target dimensions when evaluating a target dimension, i.e., inter-dimension dependence, we propose CorrGap. To measure this, CorrGap uses the difference in correlations between LLM-predicted scores and ground truth scores across different groups of texts. Using CorrGap, we show that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks. To mitigate inter-dimension dependence, we propose DimCheck, a method that iteratively removes unrelated evidence from COTs generated by LLM judges in a step-wise way. We show that DimCheck mitigates inter-dimension dependence and outperforms strong baselines across three LLMs and four tasks. We also show that smaller trained LLMs can approximate larger LLMs in DimCheck, with much lower inference costs.