Search papers, labs, and topics across Lattice.
The paper introduces BFS-PO, a reinforcement learning algorithm designed to mitigate overthinking in Large Reasoning Models (LRMs) trained with RL. BFS-PO employs a best-first search strategy with backtracking based on maximum entropy nodes to discover the shortest correct reasoning chain. Experiments on various benchmarks demonstrate that BFS-PO improves both accuracy and conciseness of LRM outputs compared to existing RL methods like GRPO/DAPO.
Stop paying for verbose overthinking: BFS-PO slashes LRM output length while simultaneously boosting accuracy.
Large Reasoning Models (LRMs) such as OpenAI o1 and DeepSeek-R1 have shown excellent performance in reasoning tasks using long reasoning chains. However, this has also led to a significant increase of computational costs and the generation of verbose output, a phenomenon known as overthinking. The tendency to overthinking is often exacerbated by Reinforcement Learning (RL) algorithms such as GRPO/DAPO. In this paper, we propose BFS-PO, an RL algorithm which alleviates this problem using a Best-First Search exploration strategy. Specifically, BFS-PO looks for the shortest correct answer using a backtracking mechanism based on maximum entropy nodes. By generating progressively shorter responses during training, BFS-PO learns to produce concise reasoning chains. Using different benchmarks and base LRMs, we show that BFS-PO can simultaneously increase the LRM accuracy and shorten its answers.
