Search papers, labs, and topics across Lattice.
This paper introduces SAFARI, a novel framework for diagnosing failures in autonomous agents that overcomes the limitations of traditional context loading methods by utilizing a tool-augmented diagnostic loop and a persistent Short-Term Memory (STM). By decoupling diagnostic accuracy from the constraints of context limits, SAFARI enables effective fault attribution even when relevant information is significantly beyond the model's native context window. Experimental results show that SAFARI achieves a 20% improvement on the Who&When dataset and maintains high precision in scenarios where conventional methods falter, demonstrating its robustness in complex multi-agent tasks.
SAFARI maintains diagnostic precision even when critical fault information lies five times beyond the model's context limits, a feat traditional methods cannot achieve.
As autonomous agents tackle increasingly complex multi-step, multi-agent tasks, their execution trajectories have scaled beyond the constraints of even the largest context windows. Current methods for effectively diagnosing agent failures load the full trajectory into an LLM's context window, which suffers from attention dilution and fails when agentic traces inevitably exceed context limits. To address this, we introduce SAFARI (Scaling long-horizon Agentic Fault AttRibution via active Investigation), a framework that replaces linear context loading with a tool-augmented diagnostic loop. By equipping LLMs with a specialized toolbox to read and search trajectory segments alongside a persistent Short-Term Memory (STM) for cross-turn reasoning, SAFARI effectively decouples diagnostic accuracy from architectural context limits. Our experiments demonstrate that SAFARI outperforms state-of-the-art results by 20% on the Who&When dataset within a 1M token budget, and by 19% on TRAIL GAIA subset on a 25K token budget. Most significantly, SAFARI maintains a 0.58 precision even when the target fault resides 5x beyond the model's native context window, a scenario where traditional evaluators fail entirely.
