Search papers, labs, and topics across Lattice.
The authors evaluate LLM faithfulness under context-memory conflicts by benchmarking text generation from factual, counterfactual, and fictional RDF triples containing regional knowledge across English, Czech, Slovak, and Upper Sorbian. Assessing whether models reject inputs that contradict parametric memory is critical for ensuring reliable retrieval-augmented generation and knowledge updating in non-English domains. Contrary to expectations, human evaluation and aligned LLM judges show that context-memory conflict is virtually negligible—reducing faithfulness scores by only 0.05 on a 1–5 scale—while revealing that unaligned LLM judges substantially overestimate this effect.
LLMs rarely ignore context simply because it contradicts their parametric memory, meaning reported context-memory conflicts may largely be artifacts of poor LLM-as-a-judge selection.
Large language models (LLMs) are prone to hallucinating or misinterpreting facts, which impairs their usability in retrieval-augmented generation or data-to-text systems. We analyse how faithfulness of LLMs to provided context depends on how plausible they perceive the context to be (context-memory conflict). To better identify error patterns, we make use of the increased difficulty of non-English and low-resource language text generation and input data based on local knowledge, only partially captured in models'parametric knowledge. We let the models generate text in English, Czech, Slovak and Upper Sorbian from factual (FA), counterfactual (CFA) and fictional (FI) RDF triples containing local Czech and Slovak data. Contrary to our expectations, we observe only a weak context-memory conflict on the human-annotated sample. For Kimi K3 as an LLM judge, which agrees well with human annotations on the sample, counterfactual inputs receive only slightly lower faithfulness scores than factual ones (-0.05 on a 1-5 scale). We also find that a suboptimal choice of LLM judge would lead to overestimating the strength of the context-memory conflict.
