Search papers, labs, and topics across Lattice.
This paper introduces HiMPO, a novel framework for optimizing memory writing in long-horizon agents by addressing the credit assignment problem that arises from causally entangled memory updates. By estimating the local utility of memory updates and applying hindsight relevance as a filter, HiMPO effectively assigns less-entangled credit to memory-writing actions, leading to improved performance on both judge-based open-domain tasks and objective compressive-memory question answering. The results demonstrate that HiMPO not only enhances memory efficiency but also reduces blame leakage from tool-induced errors, thereby improving the fidelity of memory update attributions.
HiMPO assigns less-entangled credit to memory updates, significantly boosting long-horizon agent performance while minimizing blame leakage from errors.
Long-horizon agents rely on memory mechanisms to compress interaction history, but optimizing memory writing faces a distinct credit assignment challenge: a memory update may be rewarded or penalized due to downstream tool failures, noisy observations, or reasoning errors rather than its own contribution. This causally entangled credit can lead agents to discard useful evidence or preserve irrelevant information. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework for assigning less-entangled credit to memory-writing actions in long-horizon agents. HiMPO first estimates the local utility of a memory update by comparing the task-relevant information recoverable from the previous and updated memories under the same pre-write state. It then uses hindsight relevance as a bounded retrospective filter that attenuates memory credit when local utility is not supported by the target outcome. The resulting memory-specific advantage is applied only to memory tokens, while trajectory-level rewards optimize the rest of the agent behavior. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves over strong memory-based and RL-based baselines while preserving compressed-context efficiency. Controlled interventions further show that HiMPO reduces blame leakage from tool-induced errors and improves attribution fidelity of memory updates.
