Search papers, labs, and topics across Lattice.
This paper introduces AgentFootprint, a benchmark that quantifies the persistent storage footprint of LLM agents, addressing the overlooked impact of data retention on evaluation metrics. By employing a serialization-aware metric suite, the study reveals significant discrepancies in storage retention across different configurations, with findings indicating up to a 15.7x difference in retained bytes despite identical performance metrics. The results highlight the importance of considering storage metrics alongside traditional accuracy measures, as they can vary drastically and influence the overall resource efficiency of LLM agents.
Persistent storage can vary by over 15x even among LLM agents achieving identical task accuracy, revealing a hidden cost of agent evaluation.
LLM agent benchmarks measure task completion, reliability, and inference cost, but not the persistent data an agent run leaves on disk, including logs, context snapshots, checkpoints, and debug traces. We introduce AgentFootprint, a cross-framework benchmark of post-run agent storage footprint. Its serialization-aware metric suite measures total retention, channel composition, duplication, growth, compressibility, and conversation-history reconstructability. It addresses a measurement trap: naive byte-level measurement understates duplication by an order of magnitude because database paging and JSON escaping obscure repeated content. A fixed-trace control separates agent-generated logical volume from persistence-layer amplification: replaying the same trajectory through seven persisting frameworks yields a 6.7x spread. Under identical models, tools, and tasks, configurations with 100% accuracy differ by 15.7x in retained bytes, although their defaults support different recovery and audit capabilities. Three full-history configurations grow superlinearly on a repeated-observation stress task. Exported trajectories from 108 instance-normalized SWE-bench Verified submissions span three orders of magnitude per instance, with no detectable correlation with resolve rate. A content-addressed store reduces retention by 4.8x-32.7x while preserving every reconstructability score. These results establish persistent storage as a resource metric to report jointly with accuracy and reconstructability.
