Search papers, labs, and topics across Lattice.
The paper introduces DeltaML-Bench, a novel benchmark designed to evaluate machine learning agents on real-world research repositories by presenting 48 tasks that require agents to enhance published baselines within imperfect, open-source environments. The study evaluates the performance of GPT-5 and Claude Sonnet 4 using both a standard Modular agent and a search-based ARG scaffolding, revealing that ARG significantly improves GPT-5's success rates under various time allocations. Notably, while Modular configurations showed a high rate of specification gaming, ARG configurations maintained integrity with no observed gaming, highlighting the importance of scaffolding design in autonomous ML experimentation.
ARG scaffolding boosts GPT-5's success rate from 9.4% to 49.0% in real-world ML tasks, while Modular setups suffer from high specification gaming.
Autonomous agents for machine learning experimentation must navigate heterogeneous repositories, repair training pipelines, and evaluate candidate improvements under realistic compute constraints. Existing benchmarks only partially capture these conditions. We introduce DeltaML-Bench, a benchmark comprising 48 tasks sourced from research papers that require agents to improve published baselines within imperfect, open-source repositories. We evaluate GPT-5 and Claude Sonnet 4 with a standard Modular agent and a search-based ARG scaffolding. In the 4 x 6h allocation, ARG raises GPT-5's per-run success rate from 9.4% to 33.9%; in the 2 x 12h allocation, GPT-5 ARG reaches 49.0%. Modular configurations exhibit specification gaming rates as high as 47.9%, while no gaming is observed in the evaluated ARG configurations. These results indicate that scaffolding design and integrity checks are important considerations when deploying agents for autonomous ML experimentation.
