Search papers, labs, and topics across Lattice.
This paper addresses the optimization of LLM evaluation pipelines by formulating judge-panel design as a role-conditioned allocation problem, which determines the most effective judges to call for specific examples and when to stop panel construction. The proposed method utilizes a small labeled audit set to estimate the roles of judges鈥攊dentifying copies, complements, and specialists鈥攁nd generates a policy for their conditional use based on validation gains. The findings reveal a regime map that guides the strategic routing of judges, ultimately leading to a more efficient and effective evaluation process across various tasks, including reasoning and safety audits.
Routing LLM judges based on their roles can significantly enhance evaluation efficiency and effectiveness, revealing when to stop calling judges to optimize performance.
LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers. The deployment question is not only which judge is best, but which judges should be called, on which examples, and when panel construction should stop. We formulate judge-panel design as a role-conditioned allocation problem. From a small labeled audit set, declared slices, and judge costs, the method estimates target-relative roles: copies add no conditional information, complements improve the global panel, and specialists help only on slices. These roles induce a policy: drop copies, add complements globally, route specialists conditionally, and stop when validation gain falls below a threshold. Across reasoning, code, safety, preference, reward-model, summarization, and math audits, the method is compared with single judges, flat panels, matched diversity heuristics, full-call stacking, reliability juries, and frugal cascades. The result is a regime map for judge calls: route specialists on deployable slices, stop in saturated verifier regimes, keep broad ensembles when their risk benefit is worth the cost, and ignore conditional copies. The output is a reusable, auditable call plan for the next evaluation batch.
