Search papers, labs, and topics across Lattice.
This paper introduces ConflictScore, a novel metric designed to evaluate how well language models recognize and handle conflicting evidence in their responses. By decomposing model outputs into atomic claims and assessing them against grounding documents, ConflictScore provides two measures: ConflictScore-Count (CS-C) and ConflictScore-Ratio (CS-R), which quantify the presence and balance of conflicting information. Experiments demonstrate that ConflictScore not only identifies overconfident claims but also enhances model truthfulness when applied as feedback on the TruthfulQA benchmark.
ConflictScore reveals that language models often overlook conflicting evidence, leading to overconfident and inaccurate claims.
Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to capture when both supporting and contradicting evidence coexist. We introduce ConflictScore, a novel metric that quantifies how well a model's response acknowledges conflicting evidence in its grounding documents. Our framework decomposes responses into atomic claims, labels each claim against each grounding document, and then aggregates these labels into two complementary measures: ConflictScore-Count (CS-C), the proportion of claims exhibiting conflicts, and ConflictScore-Ratio (CS-R), the balance between supporting and contradicting evidence. We develop ConflictBench, a benchmark covering diverse forms of conflicts such as ambiguity, contradiction, and divergent opinions, to systematically evaluate our metric. Experiments show that ConflictScore effectively detects overconfident claims across domains and can serve as a corrective feedback mechanism that improves truthfulness on TruthfulQA.
Microsoft Research