Search papers, labs, and topics across Lattice.
The paper introduces EvoClaw, a benchmark designed to evaluate AI agents' ability to handle continuous software evolution by reconstructing verifiable Milestone DAGs from noisy commit logs using the DeepCommit pipeline. EvoClaw assesses agents on sustaining system integrity and limiting error accumulation, aspects often overlooked in existing benchmarks focused on isolated coding tasks. Experiments with 12 frontier models across 4 agent frameworks demonstrate a significant performance drop from >80% on isolated tasks to ≤38% in continuous settings, highlighting the agents' challenges with long-term maintenance and error propagation.
AI agents that ace isolated coding tasks fall apart when faced with the messy reality of continuous software evolution, dropping from 80% to 38% success rates in a new benchmark.
With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as semantically cohesive development goals. These executable sequences enable EvoClaw, a novel benchmark that requires agents to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.
Stanford HAI