Search papers, labs, and topics across Lattice.
This paper introduces IB-Score, a metric based on Information Bottleneck theory, to quantify the exploration-exploitation balance in online reinforcement learning for LLMs by measuring the trade-off between reasoning diversity and mutual information with the correct answer. They find that common online RL approaches fail to maintain this balance, leading to suboptimal performance. To address this, they propose IB-TPO, a framework that optimizes IB-Score and uses an IB-guided tree sampling strategy, achieving significant performance gains over GRPO and other state-of-the-art methods on standard benchmarks.
Online RL for LLMs suffers from an imbalanced exploration-exploitation trade-off, but IB-TPO solves this with a novel Information Bottleneck-driven tree sampling strategy that boosts performance by up to 3.6%.
Recent advances in online reinforcement learning (RL) for large language models (LLMs) have demonstrated promising performance in complex reasoning tasks. However, they often exhibit an imbalanced exploration-exploitation trade-off, resulting in unstable optimization and sub-optimal performance. We introduce IB-Score, a novel metric grounded in Information Bottleneck theory that evaluates policy's exploration-exploitation balance by quantifying the trade-off between step-level reasoning diversity and mutual information shared with the correct answer. Analysis based on IB-Score shows that popular online RL approaches (e.g., GRPO) with common regularizers fail to consistently maintain balance during training with suboptimal results. To address this, we propose Information Bottleneck-driven Tree-based Policy Optimization (IB-TPO), a principled framework that formulates IB-Score as a fine-grained optimization objective and utilizes a novel IB-guided tree sampling strategy that not only improves the efficiency of online sampling with 50% more trajectories under the same token budget, but also reuses the tree structure for effective IB-Score Monte Carlo estimation. Extensive experiments across standard benchmarks show that our method significantly outperforms GRPO baseline by 2.9% to 3.6% and also outperforms other state-of-the-art online RL approaches. Our code is available at https://github.com/alibaba/EfficientRL.
