Search papers, labs, and topics across Lattice.
This paper introduces Bayesian Manifold Curriculum (BMC), a novel framework that redefines problem sampling in reinforcement learning for large language models (LLMs) as a manifold-structured bandit problem. By organizing tasks into a hierarchical tree and leveraging Bayesian learning, the authors demonstrate that sampling strategies significantly impact learning productivity, diversity, and utility. The findings reveal that merely prioritizing problem difficulty is inadequate for optimizing downstream performance, emphasizing the need for a structured approach to task selection.
Prioritizing problem difficulty alone can undermine LLM performance, as a structured approach to sampling reveals critical trade-offs in learning efficiency and task coverage.
Reinforcement learning (RL) is a central approach for improving reasoning capabilities in large language models (LLMs), where training efficiency depends critically on how problems are sampled during optimization. Existing adaptive curriculum learning methods typically prioritize prompts of intermediate difficulty, treating problem selection as a standard bandit problem with independent arms and overlooking the structured, heterogeneous nature of the task space. In this work, we frame problem sampling as a manifold-structured bandit problem with endogenous non-stationarity: problems are related through the model's latent representation space, and sampling decisions can steer how learning signals evolve across that space. To operationalize this perspective, we introduce Bayesian Manifold Curriculum (BMC), a structure-aware framework that organizes problems into a hierarchical task tree and applies Bayesian learning to guide sampling. Empirically, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance). These results show that prioritizing difficulty alone is insufficient for strong downstream performance, highlighting the importance of incorporating structure and type-awareness into problem sampling.
