Yu Cheng

Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both effects.

LLMs can be taught to "think longer" and explore more diverse reasoning paths in-context via a simple length-incentivized reward, leading to improved generalization.