Search papers, labs, and topics across Lattice.
This paper introduces SERPO, a novel approach for test-time reinforcement learning that enables language models to self-evolve without relying on labeled feedback or answer voting, which is unsuitable for open-ended generation. SERPO employs a closed-loop system that co-evolves response evidence, query-specific rubrics, and policy parameters, facilitating a dynamic adaptation process. The results demonstrate significant performance improvements on multiple benchmarks, with up to 20.63 points gained on HealthBench and 20.31 points on ResearchQA, showcasing its effectiveness in both in-domain and out-of-domain scenarios.
SERPO achieves up to 20.63 points improvement on HealthBench by enabling language models to self-evolve their evaluation criteria in real-time without external feedback.
Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answer voting and therefore do not extend naturally to open-ended generation, where valid responses cannot be mapped to a shared canonical answer. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model's own outputs. We introduce SERPO (Self-Evolving Rubric Policy Optimization), which replaces answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters. Good-Normal-Bad (G-N-B) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that discriminate these archives; probabilistic criterion scoring converts verdict-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals. New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop. Across two model configurations, two in-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20.63 and 20.31 points over the corresponding base models, raises the six-benchmark macro-average by up to 8.06 points, and supports OOD transfer and continued cross-benchmark evolution.
DAMO