Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of hidden evidence-use forgetting in multimodal large language models (MLLMs), where models retain answer accuracy but lose grounding in the evidence used to generate those answers. The authors introduce a novel framework called RCL, which constrains reliance on evidence channels while optimizing for task learning and prediction preservation without the need for replay mechanisms. Experimental results demonstrate that RCL significantly enhances performance and reduces reliance drift across various multimodal tasks, highlighting the importance of maintaining the evidence path behind correct answers in continual learning scenarios.
Hidden evidence-use forgetting can lead to accurate answers that lack grounding, but a new framework shows how to preserve the evidence path behind those answers.
Multimodal large language models must continually adapt to evolving tasks and domains, yet standard continual learning metrics mainly measure whether old answers remain correct, leaving the stability of multimodal grounding largely unexamined. We study this overlooked failure mode and ask whether a continually adapted MLLM can preserve not only what it answers, but also how it uses visual, textual, OCR, chart, and document evidence. We identify \emph{hidden evidence-use forgetting}, where answer accuracy is retained while the model silently shifts toward different or less grounded evidence channels, and propose \textsc{RCL}, a replay-free reliance-constrained continual learning framework. \textsc{RCL} freezes the previous checkpoint as a behavioral reference, estimates teacher and student evidence-reliance profiles through counterfactual channel interventions, and jointly optimizes task learning, prediction preservation, and reliance preservation without adding inference-time cost. Across CoIN, COAST, MCITlib, and an evidence-sensitive multimodal stream, \textsc{RCL} consistently improves final performance and reduces forgetting over replay-free, PEFT, routing, and memory-assisted baselines, while substantially lowering modality reliance drift, dominant evidence flips, and hidden forgetting rates. These results suggest that robust continual multimodal learning requires preserving the evidence path behind correct answers, not merely the answers themselves.
