Search papers, labs, and topics across Lattice.
This paper introduces DG-Mem, a dual-grained memory framework designed to enhance frozen multimodal large language models (MLLMs) by integrating a non-parametric, externally stored memory that utilizes both instance-grounded exemplars and category-level schemas. By employing an online concept categorizer and a Shapley context attribution method, DG-Mem allows for dynamic memory retrieval that adapts during training and optimizes performance at test time without requiring parameter updates. The results demonstrate significant improvements in mathematical reasoning tasks across multiple benchmark models, highlighting the effectiveness of memory augmentation in enhancing MLLM capabilities.
DG-Mem achieves substantial gains in reasoning tasks by leveraging a novel memory architecture that adapts dynamically without altering model parameters.
Frontier multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning. Parameter-level adaptation is unavailable for closed-weight or on-device backbones, and stateless prompting forfeits any compounding benefit from problems already solved. We propose \textbf{DG-Mem}, a dual-grained agentic memory framework that augments a frozen MLLM with a non-parametric, externally stored memory built once from training-time rollouts and consulted read-only at test time. Motivated by the Complementary Learning Systems (CLS) account of human memory, DG-Mem factors its store into an instance-grounded exemplar memory and a category-level schema memory of IF-THEN rules, with a transient reflection store mediating their construction so that schemas are synthesized only from abstract reflections, never from exemplar text. Two design choices distinguish DG-Mem: an online concept categorizer that grows the category space incrementally during training rather than committing to a predefined taxonomy, and a Shapley context attribution procedure that decomposes correctness across the entire retrieved rule set and yields a per-rule utility that re-weights retrieval at test time. The pipeline introduces no gradient updates and is deployable on closed-weight or on-device backbones. Across MathVista, MMMU, and MMMU-Pro on four open-weight and proprietary backbones (Qwen3.5-27B, Qwen3.5-122B-A10B, GPT-5-Nano, Gemini-3-Flash), DG-Mem improves consistently over no-memory and competitive memory baselines.
