Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome the gradient starvation that occurs when fixed problem pools saturate during RL post-training, this work establishes frontier learning, an open-ended curriculum framework that generates reasoning problems online. The framework treats procedural generator parameters as a search space and optimizes a regret signal to actively sample tasks at the threshold of model capability, where rollout success-failure variance is highest. Across multiple reasoning domains and model families, dynamically synthesizing frontier-difficulty problems yields consistent relative performance improvements over standard fixed-pool GRPO baselines.
Static problem pools cause LLM reasoning post-training to stall as models saturate the dataset, but dynamically synthesizing frontier tasks via regret-guided procedural generation keeps learning signals saturated for sustained gains.
Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.