Search papers, labs, and topics across Lattice.
The paper introduces SOAR, a meta-RL framework where a teacher LLM generates synthetic problems for a student LLM to solve, with the teacher rewarded based on the student's improvement on a held-out set of hard problems. This approach addresses the challenge of training reasoning models with sparse rewards by leveraging the teacher model's latent knowledge to create a curriculum of stepping stones. Experiments on mathematical benchmarks demonstrate that SOAR unlocks learning from binary rewards, outperforms intrinsic reward schemes, and highlights the importance of structural quality in generated questions.
LLMs can escape their reasoning plateaus by learning to teach themselves, even when they can't initially solve the problems, unlocking a path to self-improvement without curated data.
Can a model learn to escape its own learning plateau? Reinforcement learning methods for finetuning large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? To explore this, we design SOAR: A self-improvement framework designed to surface these pedagogical signals through meta-RL. A teacher copy of the model proposes synthetic problems for a student copy, and is rewarded with its improvement on a small subset of hard problems. Critically, SOAR grounds the curriculum in measured student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of mathematical benchmarks (0/128 success) reveals three core findings. First, we show that it is possible to realize bi-level meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful stepping stones. Second, grounded rewards outperform intrinsic reward schemes used in prior LLM self-play, reliably avoiding the instability and diversity collapse modes they typically exhibit. Third, analyzing the generated questions reveals that structural quality and well-posedness are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to actually solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data.
