Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces MM-Mem, a novel pyramidal multimodal memory architecture inspired by Fuzzy-Trace Theory, to address the limitations of existing methods in long-horizon video understanding for multimodal large language models. MM-Mem employs a hierarchical memory structure (Sensory Buffer, Episodic Stream, Symbolic Schema) to progressively distill fine-grained perceptual traces into high-level semantic schemas. The authors introduce a Semantic Information Bottleneck objective and SIB-GRPO to optimize memory compression and task-relevant information retention, along with an entropy-driven top-down memory retrieval strategy.
Achieve state-of-the-art long-horizon video understanding by compressing multimodal memories into high-level semantic schemas, enabling efficient reasoning without losing crucial details.
While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy, which first tries with the abstract Symbolic Schema and progressively"drills down"to the Sensory Buffer and Episodic Stream under high uncertainty. Extensive experiments across 4 benchmarks confirm the effectiveness of MM-Mem on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code is available at https://github.com/EliSpectre/MM-Mem.