Search papers, labs, and topics across Lattice.
This paper introduces Adaptive Auto-Harness, a novel framework designed to enhance the deployment of LLM agents in open-ended task streams by addressing the limitations of traditional auto-harness systems. By decomposing the challenges into evolution loss and adaptation loss, the framework employs a stateful multi-agent evolver and a harness tree with solve-time routing, allowing for sustained performance improvements in dynamic environments. Experimental results demonstrate that Adaptive Auto-Harness significantly outperforms five existing baselines across various task streams, highlighting its effectiveness in adapting to shifting problem distributions and heterogeneous tasks.
Sustained self-improvement in LLM agents is achievable through a novel adaptive framework that outperforms traditional methods in dynamic task environments.
Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, a framework and system for such streams. The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss. The system addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal. Across prediction-market, security-competition, and event-forecasting streams, Adaptive Auto-Harness outperforms five existing auto-harness baselines and ablations attribute gains to better construction, routing, or targeted human steering. Code is available in https://github.com/A-EVO-Lab/AdaptiveHarness .
Amazon Science