Search papers, labs, and topics across Lattice.
This paper explores the scalability of large reasoning models (LRMs) beyond human supervision by developing a framework that transitions from human-judged rewards to autonomous verifiers and self-generated curricula. It identifies a five-level ladder that delineates the extent of human control in the learning process and highlights the associated risks, such as reward hacking and curriculum collapse. The findings underscore the potential for LRMs to autonomously improve while also outlining critical challenges that must be addressed to ensure safe and effective scaling toward superintelligence.
As human oversight wanes, LRMs could autonomously evolve, but this shift introduces significant risks like reward hacking and feedback drift.
Recent advances in large reasoning models (LRMs) have shown that reinforcement learning with verifiable rewards (RLVR) can substantially improve reasoning in mathematics and code, where outcomes can be checked automatically. Extending this progress to open-ended and agentic tasks remains difficult because reliable rewards are harder to obtain and direct human supervision cannot keep pace with the scale and complexity of model-generated experience. This paper studies how LRMs can continue to improve as human supervision gradually recedes from the learning loop. We examine two connected dimensions of this problem. The reward axis traces the development from per-instance human judgments to reusable verifiers and rewards that operate even without human feedback. The experience axis examines how learning can progress from human-curated tasks and environments toward self-generated curricula, constructed environments, and autonomous co-evolution. We connect these dimensions through a five-level ladder from L0 to L4 that identifies which parts of the learning process remain under continued human control. Our analysis further highlights the risks introduced by increasingly autonomous rewards and experience generation, including reward hacking, feedback drift, curriculum collapse, and environment errors. Consequently, we also provide the evaluation around three complementary objects: policy capability, feedback fidelity, and experience quality. This analysis provides a structured account of current approaches to scaling LRMs beyond human supervision and the open problems involved in developing self-sustaining learning systems toward superintelligence. Furthermore, we maintain a continuously updated https://github.com/visitworld123/Awesome-Scaling-LRM-Beyond-Human-Supervision{GitHub repository} to track the latest advances.
NUS