Search papers, labs, and topics across Lattice.
The paper introduces RobustVisRAG, a causality-guided dual-path framework to improve the robustness of Vision-based Retrieval-Augmented Generation (VisRAG) models under visual degradations. RobustVisRAG disentangles semantic and degradation factors by using a non-causal path to capture degradation signals and a causal path to learn purified semantics. The framework is trained with Non-Causal Distortion Modeling and Causal Semantic Alignment objectives and evaluated on a new Distortion-VisRAG dataset, demonstrating significant improvements in retrieval and generation performance under real-world degradations.
VisRAG models can now handle real-world image degradations like blur and shadows without sacrificing accuracy, thanks to a new causality-guided architecture that disentangles semantics from visual distortions.
Vision-based Retrieval-Augmented Generation (VisRAG) leverages vision-language models (VLMs) to jointly retrieve relevant visual documents and generate grounded answers based on multimodal evidence. However, existing VisRAG models degrade in performance when visual inputs suffer from distortions such as blur, noise, low light, or shadow, where semantic and degradation factors become entangled within pretrained visual encoders, leading to errors in both retrieval and generation stages. To address this limitation, we introduce RobustVisRAG, a causality-guided dual-path framework that improves VisRAG robustness while preserving efficiency and zero-shot generalization. RobustVisRAG uses a non-causal path to capture degradation signals through unidirectional attention and a causal path to learn purified semantics guided by these signals. Together with the proposed Non-Causal Distortion Modeling and Causal Semantic Alignment objectives, the framework enforces a clear separation between semantics and degradations, enabling stable retrieval and generation under challenging visual conditions. To evaluate robustness under realistic conditions, we introduce the Distortion-VisRAG dataset, a large-scale benchmark containing both synthetic and real-world degraded documents across seven domains, with 12 synthetic and 5 real distortion types that comprehensively reflect practical visual degradations. Experimental results show that RobustVisRAG improves retrieval, generation, and end-to-end performance by 7.35%, 6.35%, and 12.40%, respectively, on real-world degradations, while maintaining comparable accuracy on clean inputs.
Microsoft Research