Search papers, labs, and topics across Lattice.
The paper addresses the problem of excessive and unnecessary reflection in Large Reasoning Models (LRMs) that leads to increased token consumption and computational overhead without improving accuracy, especially in smaller models. To mitigate this, they propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a reinforcement learning framework that dynamically balances reasoning efficiency and solution accuracy by introducing reflection and length penalties. Experiments on mathematical reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and 7B models demonstrate that ARLCP achieves a superior efficiency-accuracy trade-off, reducing response length by up to 53.1% while improving accuracy by up to 5.8%.
Smaller reasoning models can achieve both higher accuracy and shorter reasoning chains by adaptively penalizing unnecessary reflections and coordinating length penalties with problem complexity.
Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning tasks by employing test-time scaling. However, they often generate over-long chains-of-thought that, driven by substantial reflections such as repetitive self-questioning and circular reasoning, lead to high token consumption, substantial computational overhead, and increased latency without improving accuracy, particularly in smaller models. Our observation reveals that increasing problem complexity induces more excessive and unnecessary reflection, which in turn reduces accuracy and increases token overhead. To address this challenge, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a novel reinforcement learning framework designed to dynamically balance reasoning efficiency and solution accuracy. ARLCP introduces two key innovations: (1) a reflection penalty that adaptively curtails unnecessary reflective steps while preserving essential reasoning, and (2) a length penalty calibrated to the estimated complexity of the problem. By coordinating these penalties, ARLCP encourages the model to generate more concise and effective reasoning paths. We evaluate our method on five mathematical reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-7B models. Experimental results show that ARLCP achieves a superior efficiency-accuracy trade-off compared to existing approaches. For the 1.5B model, it reduces the average response length by 53.1% while simultaneously improving accuracy by 5.8%. For the 7B model, it achieves a 35.0% reduction in length with a 2.7% accuracy gain. The code is released at https://github.com/ZeweiYu1/ARLCP .
NVIDIA