Tse-Yu Wu

National Taiwan University, T‖Zsem,ideg−Zsem,iclean‖22.\mathcal{L}_{\text{FSAL}}=\frac{1}{T}\sum_{i=1}^{T}\big\|Z_{\text{sem},i}^{\text{deg}}-Z_{\text{sem},i}^{\text{clean}}\big\|_{2}^{2}. (15) The overall CSA objective is given by: ℒCSA=ℒSIL+λFSAL​ℒFSAL,\mathcal{L}_{\text{CSA}}=\mathcal{L}_{\text{SIL}}+\lambda_{\text{FSAL}}\mathcal{L}_{\text{FSAL}}, (16) where λFSAL\lambda_{\text{FSAL}} balances global and local alignment. By jointly optimizing ℒCSA\mathcal{L}_{\text{CSA}} and ℒNCDM\mathcal{L}_{\text{NCDM}}, the encoder learns to causally disentangle semantics from degradations within a single forward pass, allowing ZsemZ_{\text{sem}} to approximate the interventional representation P​(Zsem∣d​o​(D=d0))P(Z_{\text{sem}}\mid do(D=d_{0})). Overall Objective. RobustVisRAG employs two vision encoders for retrieval and generation, both trained under the same causality-guided framework. For retrieval, the encoder is optimized end-to-end with a standard contrastive objective: ℒRet=−log⁡exp⁡(⟨q,X+⟩/τ)exp⁡(⟨q,X+⟩/τ)+∑x−exp⁡(⟨q,X−⟩/τ),\mathcal{L}_{\text{Ret}}=-\log\frac{\exp(\langle q,X^{+}\rangle/\tau)}{\exp(\langle q,X^{+}\rangle/\tau)+\sum_{x^{-}}\exp(\langle q,X^{-}\rangle/\tau)}, (17) where X+X^{+} and {X−}\{X^{-}\} denote the positive and negative visual documents to query qq, and τ\tau is a temperature. The total retrieval loss combines contrastive learning with causality-guided objectives: ℒRetrieval=ℒRet+λ1​ℒCSA+λ2​ℒNCDM,\mathcal{L}_{\text{Retrieval}}=\mathcal{L}_{\text{Ret}}+\lambda_{1}\mathcal{L}_{\text{CSA}}+\lambda_{2}\mathcal{L}_{\text{NCDM}}, (18) where λ1\lambda_{1} and λ2\lambda_{2} balance causal alignment and degradation modeling. For generation, the language model remains frozen while the visual encoder is fine-tuned using only the causality-guided objectives: ℒGeneration=ℒCSA+λ3​ℒNCDM,\mathcal{L}_{\text{Generation}}=\mathcal{L}_{\text{CSA}}+\lambda_{3}\mathcal{L}_{\text{NCDM}}, (19) This training allows the adapted encoder to maintain stable semantics under degraded conditions. Through these unified optimization strategies, RobustVisRAG enforces structural intervention to suppress the spurious route D→Z→AD\!\to\!Z\!\to\!A while preserving the causal path S→Zsem→AS\!\to\!Z_{\text{sem}}\!\to\!A. Inference. At test time, we only require degradation-invariant semantics for retrieval and generation. Since the causal branch already produces ZsemZ_{\text{sem}} that has been trained to remove degradation information under the guidance of ZdegZ_{\text{deg}}, we discard the non-causal branch and feed only ZsemZ_{\text{sem}} to the downstream VisRAG modules. Thus, the inference-time computation and architecture remain compatible with the standard VisRAG pipeline, while enjoying improved robustness to visual degradations. 3.3 Distortion-VisRAG Dataset We construct the Distortion-VisRAG (DVisRAG) dataset to evaluate the robustness of Vision-based RAG pipelines under degraded visual conditions.