Search papers, labs, and topics across Lattice.
This paper introduces CLI-Universe, a synthesis engine designed to create high-quality, executable tasks for terminal agents by sampling across a multi-dimensional capability taxonomy and grounding tasks through evidence-guided research. The method ensures rigorous supervision by validating tasks in Dockerized environments and employing a multi-stage verification pipeline, resulting in the retention of only genuinely challenging and verifiable tasks. Fine-tuning the Qwen3-32B model on the newly generated CLI-Universe-6K dataset achieves a state-of-the-art performance on Terminal-Bench 2.0, highlighting the efficiency of structured task synthesis over traditional methods.
Fine-tuning a model on rigorously synthesized tasks can outperform larger models by leveraging high-fidelity data, achieving a new benchmark in terminal agent performance.
While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.
