Search papers, labs, and topics across Lattice.
This paper investigates how large language models (LLMs) manage conflicts within contextual knowledge, introducing a taxonomy of six types of conflicts and a new dataset, ContextConflict, comprising 5,781 samples. Experiments reveal that current LLMs struggle to resolve these conflicts effectively, with a notable bias towards earlier evidence that hampers their performance. A novel training-free steering method is proposed, which enhances conflict resolution by promoting a more balanced integration of evidence, resulting in improved accuracy on reasoning tasks and higher-quality summaries.
LLMs exhibit a surprising bias towards earlier evidence, which significantly impairs their ability to resolve contextual knowledge conflicts.
Most prior works focused on conflicts between an LLM's internal parametric knowledge and externally provided context. In contrast, we investigate how LLMs handle conflicts that arise within contextual knowledge itself. We introduce a taxonomy of six types of contextual conflicts (factual, inferential, temporal, granularity, perspective, and ambiguity) and contribute a comprehensive dataset ContextConflict for this setting. The dataset contains 5,781 samples, covers both reasoning and summarization tasks, and includes both explicit contradictions and implicit conflicts that require multi-step reasoning. Experiments on nine LLMs show that current models still fall short in resolving contextual knowledge conflicts. We further provide mechanistic interpretability insights into how LLMs process such conflicts, revealing their latent awareness of conflicts and the representational geometry underlying conflict processing. In addition, our analysis uncovers a consistent model bias towards earlier evidence, and this positional preference serves as a key obstacle to effective conflict resolution. Motivated by these findings, we further propose a simple training-free, label-free steering method that steers activations to encourage a more comprehensive incorporation of evidences for better conflict resolution. On our dataset, the method consistently improves accuracy on reasoning tasks and generates higher-quality, more balanced summaries for summarization tasks.
