Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Mid-Harness, a method to improve the reliability of terminal agent actions by sampling and verifying candidate actions at the model-harness boundary before execution. The research demonstrates that a capable verifier, even using the same generator model, significantly improves action reliability and trajectory success by exploiting better alternatives, outperforming simple action sampling. Specifically, using a GPT-5.6 Sol verifier with 8 sampled actions on TerminalBench-Lite boosts Pass@1 from 50.00% to 68.03%, and further improvements are achieved by distilling responses from a stronger verifier.
A single capable verifier can dramatically improve terminal agent action reliability, achieving higher success rates at lower costs than simply generating more trajectories.
Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.