Search papers, labs, and topics across Lattice.
ProEval is introduced as a proactive evaluation framework for generative AI that uses Gaussian Processes (GPs) trained via transfer learning to estimate performance scores and identify failure cases. By framing performance estimation as Bayesian quadrature (BQ) and failure discovery as superlevel set sampling, the method actively selects informative inputs for testing. Experiments across reasoning, safety, and classification benchmarks show ProEval achieves comparable accuracy to baselines with 8-65x fewer samples and reveals more diverse failure cases.
Generative AI evaluation can be sped up by 8-65x without sacrificing accuracy by proactively focusing on the most informative test cases using a pre-trained Gaussian Process surrogate model.
Evaluating generative AI models is increasingly resource-intensive due to slow inference, expensive raters, and a rapidly growing landscape of models and benchmarks. We propose ProEval, a proactive evaluation framework that leverages transfer learning to efficiently estimate performance and identify failure cases. ProEval employs pre-trained Gaussian Processes (GPs) as surrogates for the performance score function, mapping model inputs to metrics such as the severity of errors or safety violations. By framing performance estimation as Bayesian quadrature (BQ) and failure discovery as superlevel set sampling, we develop uncertainty-aware decision strategies that actively select or synthesize highly informative inputs for testing. Theoretically, we prove that our pre-trained GP-based BQ estimator is unbiased and bounded. Empirically, extensive experiments on reasoning, safety alignment, and classification benchmarks demonstrate that ProEval is significantly more efficient than competitive baselines. It requires 8-65x fewer samples to achieve estimates within 1% of the ground truth, while simultaneously revealing more diverse failure cases under a stricter evaluation budget.
DeepMind