Search papers, labs, and topics across Lattice.
This paper introduces RADAR, a diagnostic framework designed to analyze the interdependencies among evaluation criteria in rubric-based LLM-as-judge systems. By generating synthetic probes and scoring them across multiple criteria, RADAR identifies behavioral coupling that can distort aggregate evaluation scores, which is crucial for making informed decisions in model releases and updates. The framework was validated across three industry-relevant settings, demonstrating a strong correlation with human evaluations (Pearson r > 0.84) and offering actionable insights into redundancy and sensitivity in scoring.
Evaluation criteria in LLM-as-judge systems are often interdependent, and RADAR reveals these hidden couplings that can skew decision-making processes.
Rubric-based LLM-as-judge pipelines often assume that evaluation criteria provide independent signals. In practice, however, criteria can be behaviorally coupled: improving one criterion may systematically change scores on another, distorting aggregate scores used in model-release or product-update decisions. We introduce RADAR, a lightweight preflight diagnostic framework for estimating such coupling before large-scale evaluation. Given a rubric, RADAR generates targeted synthetic probes, scores each probe on all criteria, and produces a directional coupling matrix that shows which criteria co-score and how. We validate RADAR on three industry-relevant evaluation settings: NVIDIA HelpSteer2, SumPubMed, and the Yale-Salesforce SummEval benchmark. Using only a small number of probes per criterion, RADAR recovers human inter-criterion correlation structure (Pearson r > 0.84) and provides practitioners with concrete audit signals about redundancy, hierarchy, and aggregation sensitivity before committing to large-scale judging.
Microsoft Research