Search papers, labs, and topics across Lattice.
This paper systematically evaluates seven advanced autonomous agents across 36 long-horizon tasks using a novel framework that assesses their behavior through Solution Framing, Execution, and Feedback Control. The findings reveal that while these agents can effectively implement practical solutions, they largely function as engineering optimizers rather than fully autonomous researchers, exhibiting significant variability in performance and a reliance on established techniques. Key insights include the identification of process bottlenecks and the dual nature of experience reuse, which can either enhance or hinder decision-making in subsequent tasks.
Current autonomous agents excel at practical problem-solving but often lack true methodological innovation, revealing critical gaps in their development as independent researchers.
Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.
