Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces V-DyKnow, a new benchmark to evaluate how well Vision-Language Models (VLMs) handle time-sensitive factual knowledge. They benchmarked several VLMs, finding that they often output outdated facts, especially when relying on visual inputs, and that current knowledge editing and RAG methods struggle to consistently update knowledge across modalities. The benchmark and analysis provide insights into the limitations of VLMs in acquiring and updating time-sensitive knowledge.
VLMs struggle to keep up with the times, frequently serving up outdated facts, especially when interpreting images, revealing a critical gap in their real-world applicability.
Vision-Language Models (VLMs) are trained on data snapshots of documents, including images and texts. Their training data and evaluation benchmarks are typically static, implicitly treating factual knowledge as time-invariant. However, real-world facts are intrinsically time-sensitive and subject to erratic and periodic changes, causing model predictions to become outdated. We present V-DyKnow, a Visual Dynamic Knowledge benchmark for evaluating time-sensitive factual knowledge in VLMs. Using V-DyKnow, we benchmark closed- and open-source VLMs and analyze a) the reliability (correctness and consistency) of model responses across modalities and input perturbations; b) the efficacy of knowledge editing and multi-modal RAG methods for knowledge updates across modalities; and c) the sources of outdated predictions, through data and mechanistic analysis. Our results show that VLMs frequently output outdated facts, reflecting outdated snapshots used in the (pre-)training phase. Factual reliability degrades from textual to visual stimuli, even when entities are correctly recognized. Besides, existing alignment approaches fail to consistently update the models' knowledge across modalities. Together, these findings highlight fundamental limitations in how current VLMs acquire and update time-sensitive knowledge across modalities. We release the benchmark, code, and evaluation data.