Rong-Xi Tan

LLM agents significantly outperform numerical optimizers in four out of five black-box optimization domains, but adding more numerical tools doesn't always help.

Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both effects.