Search papers, labs, and topics across Lattice.
This paper introduces EM^2Mem, an event-centric multimodal memory framework that organizes heterogeneous evidence around event anchors to enhance long-video question answering. By aligning multimodal records with temporal context and semantic facts, EM^2Mem enables more efficient and effective retrieval of relevant information during inference. The framework demonstrates significant improvements in accuracy across multiple benchmarks, achieving up to 3.7 points higher than existing memory baselines while drastically reducing latency and inference token usage.
Event-centric memory can boost long-video QA accuracy by over 3 points while slashing inference costs by nearly 64%.
Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).
