Search papers, labs, and topics across Lattice.
This paper addresses the challenge of evidence conflict in retrieval-augmented generation (RAG) systems, particularly in multilingual contexts where Chinese and English evidence may contradict each other. The authors introduce the X-RAMDocs-ZHEN benchmark to diagnose these conflicts and present the X-MADAM-RAG pipeline, which effectively handles evidence through a structured approach, achieving high accuracy on the benchmark. However, the study reveals that while X-MADAM-RAG outperforms a baseline in controlled settings, it struggles significantly under stress tests that eliminate explicit answer templates, highlighting the limitations of document-level extraction in RAG systems.
X-MADAM-RAG achieves impressive accuracy in handling contradictory evidence in multilingual RAG systems, but its performance falters under stress tests, exposing critical weaknesses in document-level extraction.
Retrieval-augmented generation (RAG) systems may receive evidence that is not merely noisy but mutually contradictory. This issue becomes particularly salient in multilingual settings, where retrieved Chinese and English evidence may support incompatible answer candidates. We study this problem through X-RAMDocs-ZHEN, a controlled Chinese-English benchmark derived from RAMDocs for diagnosing evidence conflict in RAG. The benchmark contains 300 examples across six balanced conditions, including monolingual support, bilingual agreement, reversed conflict directions, and conflict with optional noise. We further examine X-MADAM-RAG, an interpretable pipeline that decomposes evidence handling into per-document candidate extraction, visible-evidence repair, deterministic candidate grouping, and conflict-aware aggregation. On the original controlled benchmark with Qwen2.5-7B-Instruct, X-MADAM-RAG achieves 0.9667 strict accuracy and 0.9767 conflict-aware success, outperforming an evidence-normalized single-call baseline. However, a zero-call rule-only extractor reaches 1.0000 on the same benchmark, revealing strong template regularity. To probe this limitation, we construct a deterministic naturalized stress test that removes explicit answer templates while preserving candidate strings. On its 100-sample subset, rule-only extraction falls to 0.0000, but X-MADAM-RAG also drops to 0.3000 strict accuracy, below both naive and evidence-normalized baselines. A privileged oracle remains perfect, indicating that document-level extraction is the main bottleneck. These findings position X-RAMDocs-ZHEN and X-MADAM-RAG as diagnostic tools for controlled evidence conflict rather than as evidence of general hallucination detection or robustness to natural retrieval.
