Search papers, labs, and topics across Lattice.
This paper introduces Task-CoEvolve, a method for optimizing large language model (LLM) harnesses through adaptive validation task selection, which significantly reduces evaluation costs. By focusing on tasks that provide the most informative feedback and adapting the sampling distribution as the harness evolves, Task-CoEvolve achieves substantial performance gains without requiring updates to the model weights. Experiments demonstrate that this approach not only reduces the number of evaluations by 80% but also matches the performance of traditional full-set evaluations, showcasing its efficiency and effectiveness.
Task-CoEvolve slashes evaluation costs by 80% while maintaining performance parity with full-set validation in LLM harness optimization.
We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.
