Search papers, labs, and topics across Lattice.
This paper introduces "Rubrics on Trial," a novel framework that generates and evolves rubrics for evaluating large language models (LLMs) using only a single query and synthetic pairwise evidence, without relying on external annotations. The approach validates each proposed rubric to ensure it effectively distinguishes answer quality and avoids common pitfalls such as over-specificity or style bias. Experimental results across five benchmark suites show that this method achieves superior accuracy and outperforms existing approaches in six out of seven evaluation sets.
Evolving rubrics from a single query can dramatically enhance LLM evaluation by eliminating reliance on external annotations and improving answer quality discrimination.
Rubrics provide structured, fine-grained signals for training and evaluating large language models (LLMs). Yet reliable query-specific rubrics are difficult to construct. Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses. Direct query-to-rubric generation avoids these resources, but provides no explicit check that a plausible rubric is useful. Such a rubric may fail to distinguish answer quality, reward an optional style, or penalize a valid alternative strategy. We introduce Rubrics on Trial, a query-only framework that evolves a rubric set from an empty set without external annotations or model training. It derives supervision solely from synthetic rubric-conditioned response pairs and validates each proposed rubric before adding it, screening out non-discriminative, over-specific, and style-only candidate rubrics. Experiments across five preference benchmark suites demonstrate the effectiveness of Rubrics on Trial, which achieves the best average accuracy and leads on six of seven evaluation sets.
