Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ACGM, a learned graph-memory retriever that constructs task-adaptive relevance graphs over agentic multi-modal web interaction histories. ACGM uses policy-gradient optimization to adapt relevance based on the evolving task state and modality, capturing heterogeneous temporal dynamics with modality-specific decay. Experiments across WebShop, VisualWebArena, and Mind2Web demonstrate that ACGM significantly improves retrieval quality, achieving 82.7 nDCG@10 (+9.3 over GPT-4o) and 89.2% Precision@10 (+7.7) compared to strong baselines.
Forget fixed-capacity buffers: ACGM learns to build task-adaptive relevance graphs over multi-modal web histories, outperforming GPT-4o by 9.3 nDCG@10 in retrieval quality.
Retrieving relevant observations from long multi-modal web interaction histories is challenging because relevance depends on the evolving task state, modality (screenshots, HTML text, structured signals), and temporal distance. Prior approaches typically rely on static similarity thresholds or fixed-capacity buffers, which fail to adapt relevance to the current task context. We propose \textbf{ACGM}, a learned graph-memory retriever that constructs \emph{task-adaptive} relevance graphs over agent histories using policy-gradient optimization from downstream task success. ACGM captures heterogeneous temporal dynamics with modality-specific decay (visual decays $4.3\times$ faster than text: $\lambda_v{=}0.47$ vs.\ $\lambda_x{=}0.11$) and learns sparse connectivity (3.2 edges/node), enabling efficient $O(\log T)$ retrieval. Across WebShop, VisualWebArena, and Mind2Web, ACGM improves retrieval quality to \textbf{82.7 nDCG@10} (+9.3 over GPT-4o, $p{<}0.001$) and \textbf{89.2\% Precision@10} (+7.7), outperforming 19 strong dense, re-ranking, multi-modal, and graph-based baselines. Code to reproduce our results is available at{\color{blue}\href{https://github.com/S-Forouzandeh/ACGM-Agentic-Web}{Saman Forouzandeh}}.