Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ToolGate, an executable acceptance pipeline designed to streamline the construction of tool-dependent scientific benchmarks by automating candidate generation and validation. The method employs three rigorous gates: executable solution verification, randomized no-tool screening, and a time-constrained tool-using agent evaluation, ensuring that only valid and non-trivial candidates are retained. In a case study with FEniCSx, ToolGate successfully filtered 500 generated items down to 128 unique, validated benchmarks, demonstrating a significant reduction in manual labor while maintaining high-quality standards.
ToolGate transforms the benchmark creation process by automating validation, allowing researchers to focus on domain design rather than repetitive checking.
Scientific benchmarks are commonly built by domain experts who write tasks and cross-check one another's work, or who adapt existing material from textbooks, published papers, and online resources. These routes can produce strong evaluations, but they require substantial per-item labor. Language models can reduce this repeated work by proposing candidates quickly. The remaining problem is acceptance. We target scientific questions whose answers require computations with specialist software rather than unaided reasoning alone. A candidate is invalid if its script fails or returns a different answer, or trivial if a model answers it without the software. We present ToolGate, which treats every generated item as a proposal and keeps it only if three gates pass. First, an executable solution script must reproduce the proposed answer when run with the scientific software. Second, randomized no-tool screening rejects candidates that models can already solve from the prompt alone. Third, a tool-using agent must solve each survivor within a fixed time limit. We instantiate ToolGate in FEniCSx with 500 generation attempts. The local-verification gate retains 478 candidates. For final reporting, we rescreen this pool after generation: two randomized no-tool screens exclude 222 from the reported pool, and direct GPT-5.5 API calls at medium reasoning (the API default) exclude another 121. Of the remaining 135, a GPT-5.5 Codex CLI agent with access to FEniCSx solves 130; exact deduplication leaves 128 unique protocol survivors. ToolGate turns repeated answer checking and difficulty screening into an auditable process while leaving domain design and final review to experts.