Search papers, labs, and topics across Lattice.
This study investigates the impact of automatic speech recognition (ASR) errors on retrieval-augmented generation (RAG) systems, specifically focusing on two enhancements: entity-graph linking and iterative reformulation. The findings reveal that while these richer configurations improve absolute F1 scores, they significantly amplify ASR errors, with the F1 gap between clean text and high-word error rate (WER) accents increasing by 36-67% compared to naive dense retrieval. The primary issue identified is the corruption of query entities, which accounts for the majority of performance degradation across multiple benchmarks.
ASR errors can dramatically worsen the performance of advanced retrieval-augmented generation systems, with error amplification reaching up to 67%.
Speech-based applications pass spoken queries through automatic speech recognition (ASR) before any retrieval module, so ASR errors enter the pipeline as a fixed upstream constraint. We empirically test whether two extensions to standard retrieval-augmented generation (RAG), entity-graph linking and iterative reformulation, absorb or amplify these errors. Using four English accents synthesized through neural TTS, we evaluate four RAG configurations on three multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA and MuSiQue) against a clean-text oracle. Although the structurally richer configurations generally retain higher absolute F1 under ASR input, both extensions amplify the error: the F1 gap from clean text to the highest-WER accent is 36-67% larger under their combination than under naive dense retrieval, on all three benchmarks. The dominant failure mode is corruption of one or more query entities, accounting for 87-96% of degradation cases on 2WikiMultiHopQA across all four methods. Two lightweight surface-form mitigations leave most of the gap intact, indicating that downstream retrieval structure amplifies remaining entity errors. We release code and data at https://github.com/ZhenghuaBao/spoken-multihop-rag .
