Search papers, labs, and topics across Lattice.
This paper investigates why reinforcement learning (RL) post-training generalizes better than supervised fine-tuning (SFT) in large language models (LLMs) by analyzing feature-level changes during training. They align internal activations across RL- and SFT-tuned models within a shared feature space and track feature evolution. The key finding is that RL induces more restrained and continually evolving feature changes that preserve base model representations, while SFT introduces specialized features that stabilize early; a compact set of task-agnostic features mediate RL generalization.
RL's superior generalization isn't about brute force, but about carefully sculpting a few key features while preserving the base model's knowledge, unlike SFT's rapid specialization.
Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models'representations. Focusing on samples where RL succeeds but the base model fails, we identify a compact, task-agnostic set of features that directly mediate generalization across diverse tasks. Feature-level interventions confirm their causal role: disabling these features significantly degrades RL models'generalization performance, while amplifying them improves base models'performance. The code is available at https://github.com/danshi777/RL-generalization.
