Wei-Jie Liu

Exposing the critic to both successful and failed same-prompt rollouts as contrastive foils resolves PPO's notoriously brittle credit assignment in multi-step reasoning—retaining its efficacy even when paired with a downscaled critic.

RL gains in LLMs are surprisingly localized to a low-dimensional, non-infinitely compressible manifold, with control directions orthogonal to principal activation components.

Pass-rate-1 prompts got you down? Composition-RL boosts LLM reasoning by automatically composing multiple problems into new verifiable questions, making better use of your existing data.

Students can surpass their teachers in on-policy distillation by extrapolating rewards and merging knowledge from domain experts, challenging the conventional wisdom that students are inherently limited by their teachers' capabilities.