Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome experimental hacking and strategy lock-in in autonomous model self-improvement, the authors created RSI-Master, a framework that couples a regularized experiment execution sandbox with reviewer-guided orchestration over a dynamic research DAG. This structured exploration enables agents to systematically navigate post-training design spaces without human oversight or reliance on evaluation shortcuts. Evaluated on PostTrainBench with Qwen3-4B-Base, the approach achieves 54.49 versus 46.53 for the top baseline at a 0.0% hacking rate, and when scaled to a 35B model, autonomously produces a checkpoint that outperforms human-tuned Instruct baselines on LiveCodeBench-v6 (41.21 vs. 37.36) and SciCode.
Autonomous model post-training can outperform human-crafted instruct baselines on hard coding benchmarks while completely eliminating reward hacking once experimental exploration is governed by a sandboxed OS and a dynamic peer-review DAG.
Recursive self-improvement (RSI) seeks to enable AI systems to participate in improving their own capabilities. A concrete pathway is autonomous model development, where agents iteratively explore post-training strategies to improve a base model. This setting faces two challenges: agents may exploit open-ended experimental actions through hacking, and repeated experimentation may lead to strategy lock-in, where an early direction is refined rather than reconsidered. We introduce RSI-Master, which addresses the two challenges at two levels: regularize step-wise actions, avoiding hacking behaviors, and promote well-structured exploration of research directions, avoiding strategy lock-in. RSI-Master consists of an Experiment OS, which enables regularized experimental actions and maintains persistent, traceable experimental records, and Reviewer-Guided Research Orchestration, which organizes Workers and Reviewers in a dynamically growing research DAG. Workers explore diverse research directions and Reviewers compare evidence across related experiments for subsequent explorations. On PostTrainBench with Qwen3-4B-Base, it averages 54.49 versus 46.53 for the strongest agent baseline, with a 0.0\% hacking rate. Scaling to 35B model, RSI-Master surpasses the human-developed Instruct model on LiveCodeBench-v6 (41.21 vs. 37.36) and SciCode, and reaches a nonzero score on HorizonMath, a benchmark of unsolved research problems on which most frontier models score near zero.
CMU ML