Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Target Policy Optimization (TPO) decouples policy improvement in RL by first constructing a target distribution from scored completions using $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$, and then fitting the policy to this target via cross-entropy minimization. This separation allows for more stable and effective updates, particularly in sparse reward settings where standard policy gradient methods struggle. Experiments across tabular bandits, transformer sequence tasks, and large language model RLVR demonstrate that TPO matches or exceeds the performance of PG, PPO, GRPO, and DG, especially under sparse reward conditions.
Decoupling policy improvement into target distribution construction and policy fitting unlocks substantial performance gains in sparse reward RL environments.
In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy-gradient methods answer both at once, so the update can overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. We introduce \emph{Target Policy Optimization} (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$ and fits the policy to it by cross-entropy. The loss gradient on sampled-completion logits is $p^\theta - q$, which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks and substantially outperforms them under sparse reward. Code is available at https://github.com/JeanKaddour/tpo.