Search papers, labs, and topics across Lattice.
MCJudgeBench is introduced as a benchmark to evaluate LLM judges at the constraint level in multi-constraint instruction following tasks, using explicit constraint lists, per-constraint gold labels, and controlled response-side perturbations. The benchmark evaluates judge reliability across label categories (yes, partial, no) and under different prompt and response perturbations, revealing that strong overall performance doesn't guarantee reliable detection across all label categories. Results show that judges with higher correctness do not always exhibit lower inconsistency, highlighting the need for constraint-level evaluation to understand failure modes.
Even top LLM judges struggle to reliably detect violations of specific constraints in complex instructions, especially when violations are partial or absent, revealing critical blind spots in current evaluation methods.
Multi-constraint instruction following requires verifying whether a response satisfies multiple individual requirements, yet LLM judges are often assessed only through overall-response judgments. We introduce MCJudgeBench, a benchmark for constraint-level judge evaluation in multi-constraint instruction following. Each instance includes an instruction, a candidate response, an explicit constraint list, per-constraint gold labels in {yes, partial, no}, and controlled response-side perturbations. The evaluation protocol further includes evaluation prompt variants to test judge stability. We evaluate proprietary and open-source LLM judges using both correctness and inconsistency metrics, distinguishing intrinsic inconsistency under stochastic decoding from procedural inconsistency under prompt and response perturbations. Our results show that judge reliability has multiple dimensions: strong overall performance does not guarantee equally reliable detection across label categories, especially for rarer partial and no cases. Judges with higher correctness do not always have lower inconsistency. Evaluation with reasoning improves correctness but does not uniformly improve stability. These findings motivate evaluating LLM judges at the constraint level to study these failure modes.
