Search papers, labs, and topics across Lattice.
This study investigates self-evolving harnesses, which are closed-loop systems that allow agents to iteratively refine their prompts, tools, and memory based on their own performance. By analyzing the evolution of these harnesses across eight programming languages and three base models, the authors reveal that the improvements are not merely benchmark-specific adaptations but rather encode a shared abstract playbook that varies in implementation according to language-specific requirements. The findings highlight that these harnesses serve as a compensation layer addressing both the engineering demands of different languages and the limitations of the underlying models, rather than being simple, opaque adaptations to benchmarks.
Evolved harnesses encode a shared abstract playbook that adapts to language-specific challenges, revealing a nuanced interplay between model limitations and engineering demands.
Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluations, but prior work reports aggregate gains rather than analyzing what the evolved artifacts encode. It therefore remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model. We disentangle these factors by holding an evolution recipe fixed across a grid of eight programming languages (Multi-SWE-Bench) and three base models, and analyzing the resulting harnesses. The recipe routes every edit through a typed failure signal and records it as a falsifiable contract, making each modification attributable after evolution. Four findings emerge. (1)The loop improves held-out solve rates over both a minimal seed and the manually designed mini-SWE-agent scaffold in most cells, but with two null regions. (2)Gains compensate recoverable execution defects, where defect mass is near zero, and gain is near zero; which defect dominates is cell-specific. A harness closes the gap between what a policy can do and what it does. (3)Evolved harnesses share an abstract playbook across languages but instantiate it with almost disjoint language ecosystem machinery. (4)The shared core transfers and can be distilled into one universal harness, while an ecosystem margin resists both and requires native re-evolution. Together, these results recast the evolved harness as a legible compensation layer, shaped jointly by the language's engineering demands and the model's behavioral gaps, rather than an opaque benchmark-tuned scaffold.
