Search papers, labs, and topics across Lattice.
This paper introduces CalibForge, an autonomous system for synthesizing terminal tasks that leverages adversarial solver calibration to enhance task difficulty relative to solver capabilities. By employing multi-solver and contrastive calibration strategies, the system generates 5,431 calibrated tasks that significantly improve model performance on various benchmarks. The results indicate that using calibrated tasks leads to substantial gains in effectiveness, with improvements of up to 30.04 percentage points on specific evaluation metrics compared to traditional methods.
CalibForge reveals that adversarial calibration can dramatically enhance the effectiveness of training data for terminal agents, leading to unprecedented performance improvements on standard benchmarks.
Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.
