Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To tackle the "consistency gap"—where high average agent accuracy masks catastrophic multi-run variance—this work introduces a self-evolving memory framework that diagnoses unstable decision steps across repeated executions. Addressing this variance is essential for production deployment, as isolated benchmark pass rates frequently obscure whether an agent can reliably execute identical workflows multiple times. Evaluated on the AppWorld benchmark with ReAct/GPT-4.1, the framework boosts 5-of-5 consecutive run success rates by 16 percentage points on identical tasks and 13 points on similar-task transfer.
Even an agent with a 77% average pass rate fails nearly half the time when tasked with executing the same prompt five times, exposing an unspoken reliability tax that step-level trajectory debugging can largely eliminate.
Large language model (LLM)-powered agents can be accurate on average yet unreliable in production, a discrepancy that has been observed but remains largely unaddressed. When given the same task five times, a ReAct agent on the AppWorld benchmark using GPT-4.1 succeeds in all five runs only 53% of the time, even though its per-run pass rate averages 77%. We call this 24-point shortfall the consistency gap, and we argue that addressing it is a precondition for trustworthy AI agent deployment. We present a self-evolving agent framework that reduces this gap by identifying unstable, low-consistency steps in agent trajectories and converting them into episodic memory the agent can draw on in future runs. At its core is a Consistency Analyzer that pinpoints where and why a trajectory is likely to flip across executions, and a Guideline Generator that converts the diagnosis into targeted guidelines, committed to memory and injected into future agent executions on similar tasks. On AppWorld with ReAct/GPT-4.1, our framework raises the fraction of tasks that succeed in all five runs by +16 points on same-task evaluation and +13 points on similar-task generalization.