Search papers, labs, and topics across Lattice.
This study explores the scaling of zero reinforcement learning (zero RL) to a trillion parameters, addressing the limitations of previous small-scale models in eliciting complex reasoning behaviors. By implementing a stable training pipeline with optimizations like clipped importance sampling and mixed-precision control, the authors demonstrate that scaling significantly enhances sample efficiency and cognitive capabilities. Key findings reveal that the model not only achieves competitive performance on mathematical benchmarks but also develops advanced reasoning traits, such as self-verification and context anxiety, which challenge the need for hand-crafted heuristics.
Scaling zero RL to a trillion parameters reveals that models can spontaneously develop advanced cognitive behaviors, making traditional heuristics obsolete.
Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the"bitter lesson"of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.
