Search papers, labs, and topics across Lattice.
This paper introduces Evo-Bench, a benchmark designed to evaluate the intrinsic harness-evolving capabilities of large language models (LLMs) across various agent domains, addressing limitations in existing evaluations that fail to isolate harness improvements from model strength. By employing a harness-guided construction framework that incorporates auxiliary-task evolution and sensitivity-aware stratified splitting, the authors rigorously assess how well models can autonomously optimize their operating harness. The results show that top models achieve significant gains, with some nearing human-engineered baselines, although performance varies across different task types, highlighting both strengths and weaknesses in harness evolution.
Autonomous evolution of agent harnesses can yield significant performance improvements, yet struggles in specific task environments reveal critical limitations.
Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness. However, systematically benchmarking this capability remains challenging, as existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research. To address these challenges, we introduce Evo-Bench, the first benchmark designed to evaluate models'intrinsic harness-evolving capabilities across Search, Office, and General agent domains. To rigorously isolate this capability, Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization. Extensive evaluations across nine frontier and open-weight models reveal that top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines. Crucially, while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows. Furthermore, our analysis exposes critical temporal anomalies like early saturation, while demonstrating that the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.
