Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
MemPilot addresses the inefficiency of query-agnostic LLM agent memory by introducing a flexible, on-demand curation framework. This system utilizes a reinforcement-learned policy to dynamically balance retrieval from pre-computed memory with query-specific multimodal history curation by heterogeneous LLMs and VLMs. Experiments show MemPilot achieves superior performance-cost-latency trade-offs compared to existing methods across diverse multimodal agent-memory tasks.
LLM agents can now dynamically curate multimodal memory on-demand, achieving optimal performance-cost-latency trade-offs by intelligently choosing between pre-computed and runtime-generated evidence.
Memory has become integral to the LLM agent ecosystem, supporting information retention and reuse across interactions. However, most existing agent memory systems construct memory in a query-agnostic manner, which can incur unnecessary preprocessing cost and discard details that later prove essential. Recent studies have begun shifting memory processing toward runtime adaptation, but typically specialize in particular operations or fixed processing schemes, leaving flexible control over performance, cost, and latency largely underexplored. To address this challenge, we present \textbf{MemPilot}, a flexible framework that orchestrates on-demand memory curation under different performance--cost--latency preferences. Specifically, we optimize a multi-step LLM policy via reinforcement learning to iteratively choose between retrieving from query-agnostic memory and delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy jointly controls evidence amount, curation instructions, model selection, and visual access, enabling fine-grained allocation of runtime computation. To optimize this policy under competing objectives, we adapt objective-wise advantage decoupling by separately estimating each objective's advantage before aggregation. Moreover, we introduce prefix-based marginal utility estimation for fine-grained credit assignment across multi-step rollouts. Experiments on five multimodal agent-memory benchmarks demonstrate favorable performance--cost--latency trade-offs across optimization preferences, with preference sweeps yielding broader frontiers than existing trade-off-aware baselines.