Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of enabling multimodal agents to acquire and retain useful information from complex, heterogeneous data for long-horizon and lifelong tasks by introducing a two-axis analytical framework for multimodal agent memory. Axis I, Formation and Representation, categorizes how multimodal information is encoded and retained, while Axis II, Memory Management, defines three tiers of memory adaptation: static, dynamic, and self-evolving. The framework synthesizes design patterns and derives principles for effective memory formation, representation, and adaptive management, offering insights into future directions for multimodal agent memory mechanisms.
A novel two-axis framework unifies the study of multimodal agent memory, revealing distinct tiers of memory adaptation from static reuse to self-evolving policies.
Multimodal models and agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling multimodal agents to continually acquire useful information from complex, heterogeneous multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine memory-management policies through accumulated experience. We introduce a two-axis analytical framework for multimodal agent memory structured around two key questions: how multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II, Memory Management). Axis I, Formation and Representation, characterizes what information is retained and how it is encoded (e.g., modality-specific embeddings, textual and structured records, latent and parametric representations) to support subsequent reasoning and action. Axis II, Memory Management, distinguishes three tiers: T1 Static Memory reuses fixed memory; T2 Dynamic Memory updates memory through a stable policy; and T3 Self-Evolving Memory updates memory through an evolving policy. Based on this framework, we synthesize recurring design patterns and derive principles for balancing source fidelity with abstraction, maintaining temporal validity and provenance, grounding memory use in recoverable evidence, and evolving memory policies over time. Furthermore, we discuss the challenges and potential future directions for the memory mechanism of multimodal agents.
MIT CSAIL