Search papers, labs, and topics across Lattice.
This survey reframes LLM post-training as structured intervention on model behavior, categorizing methods by trajectory provenance (on-policy vs. off-policy) and behavioral roles: effective support expansion, policy reshaping, and behavioral consolidation. It argues that SFT can serve both support expansion and policy reshaping, preference-based methods are typically off-policy reshaping, and on-policy RL improves behavior on learner-generated states. The authors propose that future progress relies on coordinated system design rather than individual objective improvements.
LLM post-training isn't just about objectives; it's about strategically intervening on model behavior through support expansion, policy reshaping, and behavioral consolidation.
Post-training has become central to turning pretrained large language models (LLMs) into aligned and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objective families rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary learning regimes: off-policy learning on externally supplied trajectories, and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes behavior across stages and model transitions. This perspective yields a unified reading of major paradigms. SFT may serve either support expansion or policy reshaping, whereas preference-based methods are usually off-policy reshaping. On-policy RL often improves behavior on learner-generated states, though under stronger guidance it can also make hard-to-reach reasoning paths reachable. Distillation is often best understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress in LLM post-training increasingly depends on coordinated system design rather than any single dominant objective.
