Search papers, labs, and topics across Lattice.
This study systematically investigates the instability of reinforcement learning in Small Language Models (SLMs) ranging from 70 to 500 million parameters, identifying three key failure modes: silent LoRA parameter freezing, numerical overflow in importance ratios, and catastrophic policy collapse. By employing a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism, the authors demonstrate that their approach significantly improves the stability and performance of SLMs. The proposed capacity-headroom hypothesis reveals that PPO performance is more dependent on the quality of the supervised model and the discriminative reward signal than on model size, leading to enhanced preference win rates over existing baselines with less training data.
Small Language Models can achieve robust reinforcement learning performance by focusing on model quality and reward signal rather than sheer parameter count.
The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model ($\text{PPL}<20$) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released$^{\S}$.
