Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
To investigate whether agentic models can improve without reinforcement learning, the authors evaluate Retrospection-Only Fine-Tuning (ROFT), an online procedure where an agent attempts a task, observes execution feedback, and is trained via next-token prediction exclusively on its self-generated retrospective explanations. On SWE-bench Verified and Pro, a Qwen3.5-4B agent trained with ROFT reaches 49.2% and 26.8% solve rates in 20 updates, surpassing GRPO (48.0% and 25.3% after 40 updates) with superior sample and time efficiency and without relying on a verifier. Crucially, the approach bootstraps learning on tasks where all 64 initial rollouts failed, proving that self-explanation indirectly assigns credit and transfers directly into improved downstream policy execution.
Standard next-token prediction on an agent's self-generated explanations outperforms GRPO on SWE-bench in half the training updates—even bootstrapping on tasks where every initial rollout failed.
People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT is trained on problems with mixed successful and unsuccessful base-model attempts. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 26.8% solve rates after 20 updates without using a verifier, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. It also learns to solve individual tasks on which all 64 sampled base-model attempts failed, showing that learning can begin without any initially successful trajectories. Behavioral analyses find that ROFT indirectly assigns credit to actions, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.
Microsoft Research