Search papers, labs, and topics across Lattice.
The paper introduces MMKU-Bench, a new benchmark for evaluating multimodal models' ability to update knowledge, encompassing both previously unknown knowledge and updates to existing knowledge. The benchmark contains over 25k knowledge instances and 49k images, designed to assess cross-modal consistency in knowledge updating. Experiments using SFT, RLHF, and knowledge editing methods on MMKU-Bench reveal that SFT and RLHF suffer from catastrophic forgetting, while knowledge editing struggles with continual updating, highlighting the need for more robust knowledge updating techniques.
Multimodal models struggle to keep up with our ever-changing world: SFT and RLHF catastrophically forget, while knowledge editing falters at continual learning, as shown by the new MMKU-Bench benchmark.
As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field.
