Search papers, labs, and topics across Lattice.
This paper investigates the role of sample difficulty in Reinforcement Learning with Verifiable Reward (RLVR) for LLMs, finding a non-monotonic relationship where medium-difficulty problems yield the most significant and stable reasoning improvements. Through difficulty-wise analysis and Temporal Sparse Autoencoders (T-SAE), the study reveals that easy problems reinforce basic computation while hard problems can degrade performance unless successful trajectories are sampled. The authors then propose difficulty-adaptive strategies, including backward-reasoning reformulation and T-SAE-guided training, to improve reward density and credit assignment for hard samples.
RLVR's reasoning boost isn't just about more data – it's a Goldilocks problem: too easy and LLMs skip the reasoning, too hard and they break down, but just right and they become reasoning powerhouses.
Reinforcement Learning with Verifiable Reward (RLVR) is empirically shown to notably enhance the reasoning performance of large language models (LLMs), particularly in mathematics and programming. However, the mechanistic role of Sample Difficulty in RLVR remains poorly understood. In this paper, we investigate RLVR through the lens of difficulty-wise and one-sample analysis. We find that sample difficulty has a non-monotonic effect on RLVR: easy and medium-difficulty problems yield the strongest and most stable reasoning improvements, whereas overly hard problems often provide weak learning signals, induce degenerate behaviors such as answer repetition or skipping necessary computation, and can ultimately degrade the model's pre-existing capabilities. Beyond the obverse of response, we further analyze the model's internal feature dynamics using Temporal Sparse Autoencoders (T-SAE). Easy problems mainly reinforce direct-answer and basic-computation features while suppressing deliberative-reasoning features; hard problems activate reasoning-related features but become useful only when successful trajectories are sampled; medium-difficulty problems provide a more balanced signal, strengthening both computation and multi-step reasoning features. Motivated by these findings, we propose difficulty-adaptive strategies for hard-sample utilization, using backward-reasoning reformulation and T-SAE-guided training signals to improve reward density and credit assignment during RLVR. Overall, our results identify sample difficulty as a key factor governing both the optimization dynamics and representation evolution of RLVR.
