Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces PRECISE, an extension of Prediction-Powered Inference that combines a small set of human-labeled data with a larger set judged by a language model to produce bias-corrected estimates of ranking evaluation metrics. The method significantly reduces computational complexity for hierarchical metrics like Precision@K, achieving a 21% reduction in standard error on the ESCI benchmark. In practical applications, the framework effectively identified the best system variant, leading to a notable increase in daily sales performance.
By leveraging LLM judgments alongside human annotations, this method achieves unprecedented accuracy in ranking evaluations while reducing error margins significantly.
With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.