Search papers, labs, and topics across Lattice.
This paper introduces AttriMem, an attribution-guided process-feedback framework designed to enhance memory-construction policies in LLM agents through reinforcement learning (RL). By augmenting global outcome rewards with local rewards based on token-level contributions, AttriMem addresses the fine-grained credit-assignment bottleneck that plagues traditional RL methods. Experimental results demonstrate that AttriMem significantly outperforms existing retrieval-based, heuristic, and RL-based approaches in long-horizon dialogue question answering, while also improving generalization and stabilizing RL optimization.
Token-level attribution transforms memory learning, enabling agents to identify and leverage crucial information for better performance in complex tasks.
Effective memory is crucial for LLM agents, yet constructing it effectively remains challenging. A memory-construction policy decides what information to extract, store, update, compress, or discard as interactions accumulate. Heuristic memory methods rely on subjective, task-specific rules, which can misalign with downstream objectives and limit cross-task adaptability. RL-based methods, by contrast, learn from task feedback but mainly use outcome- or module-level rewards. These coarse signals indicate task success but cannot identify which intermediate memory contents support the final answer, creating a fine-grained credit-assignment bottleneck. However, constructing such process feedback is prohibitively difficult because intermediate memory decisions lack unique ground-truth targets, while the appropriate credit varies with the agent's uncertain reasoning trajectory and therefore cannot be specified in advance. We propose AttriMem, an attribution-guided process-feedback framework for learning memory-construction policies with RL. AttriMem augments the global outcome reward with local rewards derived from token-level contributions to the final answer. Experiments on long-horizon dialogue question answering show that AttriMem outperforms retrieval-based, heuristic, and RL-based baselines, generalizes across benchmarks and answer models, stabilizes RL optimization.
