Search papers, labs, and topics across Lattice.
This paper provides a first-principles derivation of policy optimization for language models, analyzing the objective function \( J(θ) \) through the lens of its two components: trajectory probability \( p_θ(τ) \) and reward \( R(τ) \). By mapping existing policy gradient methods, from REINFORCE to GRPO and their extensions, along these two axes, the authors reveal the rationale behind design choices and the specific failures each method addresses. The unified framework not only clarifies the landscape of LLM policy optimization but also identifies compound failures that necessitate a joint approach to trajectory and reward modifications, paving the way for future algorithm development.
A unified framework reveals that existing LLM policy optimization methods often overlook compound failures that require simultaneous adjustments to both trajectory and reward components.
Policy gradient algorithms for language models optimize the same objective $J(θ) = \mathbb{E}*{τ\sim p*θ(τ)}[R(τ)]$, which has exactly two factors: the trajectory probability $p_θ(τ)$ and the reward $R(τ)$. Every method from REINFORCE to PPO to GRPO and their descendants modifies one or both factors to address a specific failure in the preceding formulation. Existing surveys organize these methods by domain or chronology, which obscures the rationale behind each design choice and the precise location of its intervention within the gradient estimator. This survey revisits the landscape of LLM policy optimization from $J(θ)$ on first principles and uses the trajectory side, induced by $p_θ(τ)$, and the reward side, induced by $R(τ)$, as the two axes along which methods are located. It covers the path from REINFORCE and PPO to GRPO, as well as post-GRPO variants, Agentic RL, and GRPO-OPD. The resulting framework is unified, diagnostic, and extensible: it analyzes methods from a shared objective, identifies which side each method modifies and why, and applies the same trajectory and reward axes across these settings. Across these settings, the framework also exposes compound failures that no single-side fix resolves and that therefore require joint design of the trajectory side and the reward side. The boundary cases and coupled failures identified by this map mark where existing solutions run out and provide a principled starting point for designing the next generation of LLM policy optimization algorithms.
