Search papers, labs, and topics across Lattice.
This paper analyzes the length bias inherent in multiple-choice benchmarks that rank candidate completions based on conditional log-probability, revealing that normalization techniques often over-correct and favor longer answers. The authors introduce a novel scoring rule called Bayesian accuracy, which incorporates a prior over answer length to mitigate linear length effects without requiring additional computational resources. Empirical results demonstrate that Bayesian accuracy consistently outperforms both standard and length-normalized accuracy in reducing length bias across various benchmarks and few-shot scenarios.
Normalizing scores for answer length can backfire, but Bayesian accuracy offers a robust solution that reduces bias without extra computation.
Multiple-choice benchmarks that rank candidate completions by conditional log-probability suffer from a length bias: because log-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice. A common mitigation is to normalize scores by completion length, but we show empirically that this heuristic frequently over-corrects, introducing a bias toward longer answers instead. We first analyze these scoring rules, characterizing when standard and length-normalized accuracy are appropriate and how their length biases depend on the distribution of completion lengths. Motivated by this analysis, we introduce \emph{Bayesian accuracy}, a scoring rule that computes the posterior probability of each candidate under an explicit prior over answer length, thereby removing linear length effects. Bayesian accuracy is a drop-in replacement for likelihood-based multiple-choice evaluation, requires no additional forward passes, and consistently exhibits lower empirical length bias than both standard and length-normalized accuracy across benchmarks and few-shot settings.
