Search papers, labs, and topics across Lattice.
This paper introduces LEEPS, a novel Latent-Guided Explore-Exploit Prompt Sampler designed to enhance reinforcement learning with verifiable rewards (RLVR) in large language models by efficiently balancing the exploitation of informative prompts and the exploration of uncertain ones. By partitioning prompts into exploit and explore portfolios and adaptively allocating rollout budgets based on recent performance, LEEPS improves the learning signal without incurring excessive generation costs. The method demonstrates significant performance gains across multiple mathematical reasoning benchmarks, achieving up to 3.7% improvement over the strongest baseline while maintaining low overhead during training.
LEEPS achieves a remarkable 3.7% performance boost in reasoning tasks by intelligently balancing prompt exploitation and exploration, redefining efficiency in RLVR for large language models.
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6\% and 3.7\% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.
