Search papers, labs, and topics across Lattice.
This paper introduces the State Value Estimation Benchmark (SVEB) to evaluate state value estimation in LLM reinforcement learning, revealing that standard PPO critics often collapse to a coarse group-average baseline. To improve state value estimation, they propose Numca, which uses numerical spans as milestones, and Hista, which leverages LLM hidden states to average disjoint rollouts. Experiments show that both Numca and Hista improve state value estimation accuracy and enhance RL training performance without significant overhead.
Standard RL critics for LLMs are basically useless, but these two simple methods can fix them.
Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead.
