Search papers, labs, and topics across Lattice.
EVOHARNESSBENCH evaluates LLM agents under dynamic operational environments by shifting non-stationarity from the task stream to the external harness across 17 multi-stage streams featuring 802 tasks, 520 tools, 42 skills, and 62 specialist agents. This matters because production agent scaffolds are constantly updated with new tooling, yet conventional continual learning benchmarks assume a static action and capability space. The benchmark reveals that merely expanding an agent's harness triggers "harness-induced forgetting" on previously mastered tasks, while also showing that competence retention and new capability adaptation often pull in opposing directions.
Expanding an agent's harness with new tools and skills often degrades its performance on tasks it previously solved, exposing a critical failure mode of "harness-induced forgetting."
Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.
