Search papers, labs, and topics across Lattice.
This paper introduces MissDiag, a diagnostic evaluation framework designed to assess incomplete-knowledge robustness in knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG). By applying structured missingness interventions while keeping questions and gold answers fixed, MissDiag allows for a nuanced analysis of how different types of evidence loss affect system performance. The findings reveal that robustness degradation is not uniform; specifically, loss of answer-adjacent evidence leads to significant performance drops, while the removal of source-context can sometimes be neutral or even beneficial.
Incomplete-knowledge robustness isn't a one-size-fits-all issue; MissDiag reveals that the type of missing evidence dramatically influences system performance.
Knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG) aim to ground answers in explicit graph evidence, but real-world knowledge graphs are often sparse, outdated, and incomplete. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer-matching protocol. To address this gap, we propose \textbf{MissDiag}, a diagnostic evaluation framework for incomplete-knowledge robustness in KGQA and KG-RAG. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop. Experiments across multiple system families show that incomplete-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer-adjacent evidence loss produces the largest observed degradation, source-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.
