Search papers, labs, and topics across Lattice.
The authors introduce MM-Lifelong, a new multimodal video dataset spanning 181 hours designed to evaluate long-term video understanding. They identify two key failure modes of existing methods: context saturation in end-to-end MLLMs and global localization collapse in agentic baselines when dealing with sparse, long-term timelines. To mitigate these issues, they propose Recursive Multimodal Agent (ReMA), which uses dynamic memory management to iteratively update a recursive belief state, achieving state-of-the-art performance on the MM-Lifelong dataset.
Current multimodal LLMs choke on long-form video understanding, either forgetting details or getting lost in the timeline, but a new agentic architecture with dynamic memory management offers a promising fix.
While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.
