Ya-Fu Li

Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both effects.

Short-context models can achieve superior reasoning performance by leveraging long-context teacher models through innovative token alignment and training strategies.