Search papers, labs, and topics across Lattice.
The paper introduces Deep Dense Exploration (DDE) to address the challenge of effective exploration in reinforcement learning for large language models, which often suffers from either root-biased sampling or inefficient budget allocation across trivial states. DDE focuses exploration on "pivots"—deep, recoverable states within unsuccessful trajectories—to improve the discovery of high-quality trajectories. The authors instantiate DDE with DEEP-GRPO, incorporating a data-driven utility function for pivot identification, local dense resampling, and a dual-stream optimization objective, achieving superior performance on mathematical reasoning benchmarks compared to GRPO and tree-based methods.
By strategically resampling from deep, recoverable states ("pivots") within unsuccessful trajectories, DDE drastically improves LLM reinforcement learning compared to methods that oversample from the root or blindly disperse budgets.
Effective exploration is a key challenge in reinforcement learning for large language models: discovering high-quality trajectories within a limited sampling budget from the vast natural language sequence space. Existing methods face notable limitations: GRPO samples exclusively from the root, saturating high-probability trajectories while leaving deep, error-prone states under-explored. Tree-based methods blindly disperse budgets across trivial or unrecoverable states, causing sampling dilution that fails to uncover rare correct suffixes and destabilizes local baselines. To address this, we propose Deep Dense Exploration (DDE), a strategy that focuses exploration on $\textit{pivots}$-deep, recoverable states within unsuccessful trajectories. We instantiate DDE with DEEP-GRPO, which introduces three key innovations: (1) a lightweight data-driven utility function that automatically balances recoverability and depth bias to identify pivot states; (2) local dense resampling at each pivot to increase the probability of discovering correct subsequent trajectories; and (3) a dual-stream optimization objective that decouples global policy learning from local corrective updates. Experiments on mathematical reasoning benchmarks demonstrate that our method consistently outperforms GRPO, tree-based methods, and other strong baselines.
