Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to evolving evaluation metrics for self-improving LLM agents, addressing the critical issue of unreliable metrics in real-world applications. The authors propose a co-evolutionary framework, termed Double Ratchet, which integrates the development of evaluation metrics with the lifecycle management of agent skills, achieving performance retention of 88-110% compared to traditional methods. Notably, the system demonstrates robustness against metric manipulation, with independent audits ensuring the integrity of evolved outputs over pre-evolution baselines.
Evolving evaluation metrics can enhance LLM performance by up to 110% while maintaining transparency and robustness against manipulation.
Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make three claims. First, metrics can be \emph{evolved}: our metric loop searches compositions of small drawback detectors under a full evolutionary lifecycle, trained to agree with a ten-item anchored reference set, regularized by consensus over unlabeled outputs, and audited against a held-out anchor it never reads, yielding a transparent, inspectable metric rather than an opaque judge. Second, since no metric exists to beat, the yardstick is recovering what an accurate metric would have enabled, and \emph{Double Ratchet}, our co-evolution of the metric with a lifecycle-managed skill loop, does so: across code generation (MBPP+), enterprise text-to-SQL (Spider~2.0-Snow), and reference-free report generation, it retains 88--110\% of the held-out lift achieved by the same skill loop driven by ground truth or the best available rubric. Third, safety comes from anchor discipline plus outer audits: removing anchor guards collapses the metric into a vacuous detector while removing the lifecycle does not; and when evolved skills gamed the report rubric, an independent judge caught it, one detector repaired it, and a task-aware judge then preferred the evolved outputs over the pre-evolution baseline in 77\% of decided pairs. We argue this failure-expecting architecture is the right default wherever no reliable automatic verifier exists.
