Search papers, labs, and topics across Lattice.
This paper introduces EvalCEGAR, a novel approach that evolves a self-writing evaluator by leveraging counterexample-guided abstraction refinement to identify and flag defects in generated responses. By utilizing a pool of Python operators that vote on candidate answers, the method effectively narrows down the evaluation process to detect subtle discrepancies between correct and incorrect outputs. The results show that EvalCEGAR can close 15.4% of the gap in performance compared to a perfect filter on unseen tasks, demonstrating its potential to enhance evaluation metrics in challenging applications like report generation.
Self-writing evaluators can significantly improve the accuracy of response assessments by autonomously identifying defects in generated content.
Agents improve quickly against a reliable automatic metric and stall without one, and the applications that need them most, report generation among them, are the ones nobody knows how to score. Can the metric write itself? Saying what makes an answer good is hard; pointing at something wrong with one is easier, so the metric we evolve is a pool of small Python operators that each flag a candidate for one named defect, or abstain, and vote. Asking a model for operators directly does not work: 183 candidates realise only 96 distinct behaviours, from one narrow region of an enormous space. EvalCEGAR instead borrows counterexample-guided abstraction refinement from program verification. It reads the pool as an abstraction and searches for a collision, two answers the operators score identically, one correct and one not. That pair, not a prompt, is the authoring request, and when a collision defeats every attempt the loop widens what an operator may read rather than resampling. On MBPP+ and HumanEval+, a sandbox whose hidden unit tests give exact ground truth, the loop writes a 55-line operator that closes 15.4% of the gap between flagging nothing and a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of our best hand-written operator's flags. On the benchmark it never saw it matches that operator's effect exactly on a third of the flags. Six of eight runs admit such an operator and all six help out of sample; our 15 hand-written operators applied together as one filter lose accuracy. An LLM judge on the same information ties that delta on a nearly disjoint set of candidates, and charges a model call per candidate forever where the operator charges none.
