Search papers, labs, and topics across Lattice.
This paper addresses the challenges of evaluating open-ended question answering (QA) by introducing a semantic correctness framework that categorizes answers into eight distinct classes based on their correctness. The authors highlight the limitations of existing evaluation metrics, which often fail to differentiate between various types of errors such as incompleteness and contradiction. Their proposed Context-Aware Precision (CAP) metric, validated against a new benchmark of 8.8k examples, demonstrates superior performance in aligning with the semantic correctness taxonomy compared to traditional methods.
A new semantic correctness framework reveals that traditional evaluation metrics often overlook critical distinctions in answer quality, leading to misleading assessments of LLM performance.
Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.
ETH