Search papers, labs, and topics across Lattice.
The paper introduces InSight, a data selection method for reinforcement learning from verifiable rewards (RLVR) that maximizes a weighted mutual information objective to improve training efficiency. InSight models data outcomes with Bayesian latent success rates, decomposing uncertainty reduction into difficulty- and evidence-dependent components, thereby addressing the limitations of difficulty-only selection strategies. Experiments demonstrate that InSight achieves state-of-the-art performance, improving performance on planning, mathematics, and general reasoning benchmarks, while also accelerating training by up to 2.2x.
Forget difficulty-based heuristics: InSight leverages weighted mutual information to select RL training data, boosting LLM reasoning and alignment with up to 2.2x speedup.
Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.
