Search papers, labs, and topics across Lattice.
This paper introduces MemoBench, a novel benchmark designed to evaluate memory consistency in video generation models within dynamically changing environments, specifically focusing on the disappear-and-reappear paradigm. By curating 360 ground-truth clips and employing a combination of automated metrics and visual question answering (VQA) assessments, the authors provide a comprehensive framework for testing how well models can recover objects that have temporarily vanished from view. The evaluation of eight state-of-the-art models highlights significant gaps in memory consistency, revealing both insights and challenges for future research in this area.
Memory consistency in video generation models falters significantly when objects disappear, with state-of-the-art models struggling to recover updated states upon reappearance.
Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.
MIT CSAIL