Search papers, labs, and topics across Lattice.
This study evaluates the compounding effects of agent optimization in a continual-learning framework using Terminal-Bench 2.0, focusing on three optimization methods: GEPA, Meta Harness, and RELAI-VCL. While all methods initially outperform a baseline agent in a static setting, only RELAI-VCL successfully maintains and compounds optimization gains when faced with new tasks, achieving a lifelong average pass rate of 76.4%. The findings highlight that incorporating regression control into the optimization loop is crucial for ensuring generalization and preventing performance degradation on unseen tasks.
Optimization gains only compound when regression control is integrated into the agent's learning process, revealing a critical factor for effective continual learning.
Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.
