Search papers, labs, and topics across Lattice.
This paper introduces BINEVAL, a novel framework that enhances LLM evaluation by breaking down evaluation criteria into atomic binary questions, allowing for interpretable and multi-dimensional scoring. By generating fine-grained evaluation questions through a meta-prompt, BINEVAL provides transparent feedback and calibrated scores, outperforming existing baselines like UniEval and G-Eval, particularly in factual consistency assessments. The framework not only aligns closely with human judgment but also facilitates iterative prompt optimization, demonstrating its practical utility in improving LLM performance across various tasks.
BINEVAL's binary question approach not only matches human evaluations but also reveals nuanced insights that traditional methods miss, paving the way for more interpretable LLM assessments.
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores. Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output, yielding transparent question-level feedback together with calibrated overall scores. This decomposition makes evaluation easier to inspect, easier to diagnose, and directly usable for prompt improvement. Across SummEval, Topical-Chat, and QAGS, BINEVAL matches or outperforms strong baselines including UniEval and G-Eval, with especially strong results on factual consistency benchmarks such as QAGS. Beyond competitive correlation with human judgments, BINEVAL better matches human score distributions and avoids the ceiling effects common in prior LLM judges, leading to better discrimination between borderline and clearly flawed outputs. We further show that the same question-level feedback supports iterative prompt optimization, improving evaluator prompts on summarization and generation prompts on IFBench under both self-update and cross-model update settings. Overall, BINEVAL provides a task-agnostic, training-free, and interpretable evaluation framework that combines strong empirical performance with practical diagnostic and optimization value.
