Search papers, labs, and topics across Lattice.
This paper introduces ProvenAI, a framework that enhances transparency in multi-hop question answering by decomposing it into three measurable layers: answer correctness, citation fidelity, and per-document influence. By applying this framework to the HotpotQA benchmark, ProvenAI evaluates over 7,400 examples and reveals a significant citation-influence gap, where citations do not always correlate with the actual influence of sources on generated answers. The findings underscore the necessity of establishing traceable links among retrieved, cited, and influential evidence to ensure meaningful transparency in retrieval-augmented systems.
The citation-influence gap reveals that clean citations can mask the real impact of uncited sources on generated answers, challenging assumptions about citation reliability in AI outputs.
Retrieval-augmented systems routinely present citations alongside generated answers, yet a citation does not confirm that the corresponding source meaningfully shaped the output. This paper introduces ProvenAI, a framework that decomposes transparency in multi-hop question answering into three independently measurable layers: answer correctness, citation fidelity against benchmark supporting evidence, and per-document influence under leave-one-resource-out intervention. Targeting the HotpotQA distractor benchmark through a seven-stage pipeline covering data normalisation, retrieval indexing, citation-aware answer generation, attribution auditing, ablation-based influence estimation, batch evaluation, and interactive inspection, ProvenAI evaluates 7,405 validation examples drawn from a canonical corpus of 509,300 passages. The system achieves 53.53% answer accuracy alongside a mean citation-fidelity score of 71.55%, and a worked example surfaces what we call the citation-influence gap: a clean citation audit co-occurring with a profile in which one cited source registers only weak influence while seven uncited sources demonstrably shift the output. We formalise the relationship between the implemented surface proxy and a token-level KL-divergence target through a stated faithfulness condition, ground the framework in causal-mediation analysis and database-provenance theory, and discuss how the three measurement layers compose with cryptographic provenance architectures emerging in autonomous scientific discovery. ProvenAI establishes that meaningful transparency in retrieval-grounded QA requires traceable links across retrieved, cited, and behaviourally influential evidence as three distinct, independently measured layers.
