Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a comprehensive evaluation pipeline for LLM-based software systems that integrates checklist creation with learned aggregation of responses to enhance reliability. By addressing both the practical requirements and the need for consistency among LLM judges, the framework significantly improves alignment with human judgments. Empirical results show that the proposed method not only increases agreement among evaluators but also provides valuable insights through self-consistency and uncertainty predictions.
Achieving higher agreement among LLM judges while aligning evaluations with human judgments could redefine quality assurance in AI development.
LLM-based software systems increasingly require effective"evals"as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally pro- vides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.