Search papers, labs, and topics across Lattice.
This paper introduces RSIBench-Data, a benchmark designed to evaluate the data-centric research capabilities of LLM agents engaged in recursive self-improvement. The study assesses four advanced agents across various domains, revealing that while they can improve training-data strategies in 58.33% of cases, their performance is inconsistent, with 78.26% of extended searches resulting in lower final scores. Trajectory analysis highlights key patterns that contribute to successful data-centric discoveries, indicating that while agents can identify useful strategies, they struggle to consistently leverage feedback for sustained improvement.
LLM agents can enhance training-data strategies in over half of their attempts, but their inconsistency reveals critical limitations in recursive self-improvement.
Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents'research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33\% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26\% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.
