Search papers, labs, and topics across Lattice.
This paper introduces the Cognitive-structured Multimodal Agent, which enhances multimodal understanding and generation by externalizing visual information into an Episodic Visual Memory, allowing for selective reactivation of relevant episodes during reasoning. The proposed architecture includes a Perceptual Abstraction Engine, a Cognitive Retrieval Engine, and a Multimodal Executive Controller, addressing the limitations of traditional models that struggle with long-horizon dialogue due to visual token explosion and unreliable referencing. The agent achieves a remarkable 91.4% retrieval accuracy over 20-turn sessions, outperforming larger baselines while significantly reducing per-turn inference time, showcasing a more efficient approach to multimodal interaction.
Surpassing larger models, this agent achieves 91.4% retrieval accuracy in long-horizon multimodal dialogues by leveraging episodic memory for efficient context management.
Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s ->12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/
